苦労する遊び人の読書録¶

Abstract (要旨)¶

本研究では,呼吸グループを識別するために知覚的,音響的に決定された呼吸の自発音声における位置の正確さを調査した. 16名の参加者はpneumotachとマイクロフォンに接続されながら,快適な話速とラウドネスで日常の簡単な話題について話すように頼まれた. 呼吸座の位置は空気力学的な信号をベースに決定され,知覚的,音響的に決定される座に対する基準として提供された. 信号検出理論は手法の精度に使用された. 結果,ポーズ検出における最もよい精度は

1. 知覚的には少なくとも2,3人の評定者の合意に基づき,
2. 音響的には,300msのポーズ持続時間をしきい値に使用する

時に達成されることを示した. 一般に,知覚をベースにした方法は音響をベースにした方法より正確であった. 自発音声における呼吸座の知覚的決定と,音響的決定,空気力学的決定との不一致は呼吸グループの決定の方法の選択に重みをつける必要がある.

The present study investigates the accuracy of perceptually and acoustically determined inspiratory loci in spontaneous speech for the purpose of identifying breath groups. Sixteen participants were asked to talk about simple topics in daily life at a comfortable speaking rate and loudness while connected to a pneumotach and audio microphone. The locations of inspiratory loci were determined on the basis of the aerodynamic signal, which served as a reference for loci identified perceptually and acoustically. Signal detection theory was used to evaluate the accuracy of the methods. The results showed that the greatest accuracy in pause detection was achieved (1) perceptually, on the basis of agreement between at least two of three judges, and (2) acoustically, using a pause duration threshold of 300 ms. In general, the perceptually based method was more accurate than was the acoustically based method. Inconsistencies among perceptually determined, acoustically determined, and aerodynamically determined inspiratory loci for spontaneous speech should be weighed in selecting a method of breath group determination.

Intro (導入)¶

発話中,呼吸のパターンは,基本的な恒常性の呼吸と,発話に伴う様々な要求を調整するために常に変化する. 霊長類の中では,人間は音声発話に向けた洗練され,柔軟な能力が特徴的に現れる(MacLarnon & Hewitt, 1999). 発話中は,呼気の持続時間は通常,完全な呼吸サイクルのわずか9-19%を占めている(吸気+呼気,MacLarnon & Hewitt, 1999). 発話のために特徴的な呼吸パターン(速い吸気と緩やかでコントロールされた呼気)は必然的に音声の出力に呼吸に関連する構造を課す. この行動は一般にBrearhGroupとして知られており,連続するシラブルか単語が一息で発話される. 健康あるいは調子の悪い話者の最適な音声のパフォーマンスに関してBreathGroupの統制はコミュニケーションに影響され影響する側面がある. 1つの発話の言語的な特徴の運搬に対して,適切な空気工学的なパワーが存在することを確かにする発話行為の前に吸気の位置及び深さは計画される必要がある(Winkworth, Davis, Adams, & Ellis, 1995).

During speech, breathing patterns are constantly changing to balance the varying demands of an utterance with those of underlying homeostatic respiration. Among primates, humans appear unique in this refined and flexible capability for sound production (MacLarnon & Hewitt, 1999). During speech, the duration of inspiration typically represents only 9%–19% of the full breath cycle (inspiration + expiration; Loudon, Lee, & Holcomb, 1988). The characteristic respiratory pattern for speech (quick inspiration and a gradual and controlled expiration) inevitably imposes a breathrelated structure on vocal output. This structure is commonly known as the breath group, a sequence of syllables or words produced on a single breath. Management of breath groups is one aspect of efficient and effective communication, for optimum vocal performance in both healthy and disordered speakers. The location and degree of inspiration must be planned prior to the production of an utterance to ensure that there is adequate aerodynamic power for conveying the linguistic properties of an utterance (Winkworth, Davis, Adams, & Ellis, 1995).

BreathGroupの特定はしばしば,録音された音声サンプルの解析,とくに,一節の読み上げ,対話,講演においての根本的なステップになる. 呼気はプロソディやそれに関連する変数に対する事前事後に存在する発話の区切りを示す. BreathGroupの便利さは以下の研究で示されている.

1. 通常発話呼吸(Hoit & Hixon, 1987; Mitchell, Hoit, & Watson, 1996)
2. 乳児の発話の発達(Nathani & Oller 2001)
3. 自動音声認識システムや文章読み上げシステム(Ainsworth, 1973; Rieger, 2003)のような発話テクノロジーのデザイン
4. 失語症の診察やリハビリ(Che, Wang, Lu, & Green, 2011; Huber & Darling, 2011; Yorkston, 1996)

これらのいろいろなアプリケーションの共通点は一呼吸に発話されたシラブルや単語のグルーピングを特定することが必要である.

Identification of breath groups is often a fundamental step in the analysis of recorded speech samples, especially for reading passages, dialogs, and orations. Inspirations mark intervals of speech that can be subsequently examined for prosody and related variables. The usefulness of breath groups has been demonstrated in studies of (1) normal speech breathing (Hoit & Hixon, 1987; Mitchell, Hoit, & Watson, 1996), (2) the development of speech in infants (Nathani & Oller 2001), (3) design of speech technologies such as automatic speech recognition and text-to-speech synthesis (Ainsworth, 1973; Rieger, 2003), and (4) the assessment and treatment of speech disorders (Che, Wang, Lu, & Green, 2011; Huber & Darling, 2011; Yorkston, 1996). Common to these various applications is the need to identify groupings of syllables or words produced on a single breath, which is the inevitable respiratory imprint on spoken communication.

BreathGroupは3つの方法で調査された.

• 知覚的 : 発話を聞くことによって評価
• 音響的 : 一般に,ある閾値以上の無声区間かパルスの探索によって評価
• 生理学的 : 典型的には胸部がよく動くところや,発話中の通気の方向を記録して評価

生理的な方法は最も妥当性の高い方法に思われる. しかし,常に簡単に発話の研究に含めるものではないし,事前に収録したサンプルを解析するのには使えない(例えば,アーカイブされた収録など). 一方で多くの研究はBreathGroupを知覚的,音響的に特定,評価している. 知覚的,音響的方法を使用したBrethGroupの研究に関する基本的な問題はどのくらいうまくこれらの方法が生理学的な解析と相関を持つのかということである.

Breath groups have been determined in three ways: perceptually (by listening to the speech output), acoustically (usually by detecting pauses or silences that exceed a criterion threshold), and physiologically (typically by recording chest wall movements or the direction of airflow during speech). The physiologic method may be considered the gold standard; however, it is not always easily incorporated into studies of speech and cannot be used to analyze previously recorded samples (such as archival recordings) that did not employ physiological measures. Although many studies identify and evaluate breath groups perceptually and acoustically, the basic question about breath group studies using perceptual and acoustic methods is how well they correlate with physiologic analysis.

知覚的な判断は呼吸サイクルに関連する発話の特徴量の音響的な判断に基づいた婉曲的な探索方法である(Bunton, Kent, & Rosenbek, 2000; Oller & Smith, 1977; Schlenck, Bettrich, & Willmes, 1993; Wang, Kent, Duffy, & Thomas, 2005; Wozniak, Coelho, Duffy, & Liles, 1999) 知覚的,音響的両方の方法は事前に収録した発話サンプルにも適応でき,ソフトウェアやハードウェアの慎ましやかな運用のみを可能にする. 一方で多くの研究ではこれらの婉曲的な方法のどちらかを使用した Breath Group の調査が行われており,これらのアプローチの精度はテストされていない. 知覚的な方法はすべて主観的なものであり,聞き手の感覚を元にしており,音響的な方法は受け入れられるポーズに対する最低限の持続時間を特定する使用者が必要である. 音声信号のしたがって,無声部分,それはおそらくパスルであるが,この目安を超えていないものは調査されていない(Campbell & Dollaghan, 1995; Green, Beukelman, & Ball, 2004; Walker, Archibald, Cherniak, & Fish, 1992; Yunusova, Weismer, Kent, & Rusche, 2005).

Perceptual determination is an indirect detection based on auditory judgments of speech features associated with the respiratory cycle (Bunton, Kent, & Rosenbek, 2000; Oller & Smith, 1977; Schlenck, Bettrich, & Willmes, 1993; Wang, Kent, Duffy, & Thomas, 2005; Wozniak, Coelho, Duffy, & Liles, 1999). Both the perceptual and acoustic methods can be applied to previously recorded speech samples and can be accomplished with only modest investment in hardware or software. Although most studies have investigated breath groups using either of these indirect methods, the accuracy of these approaches has not been tested; the perceptual method is entirely subjective, based on listeners’ impressions; the acoustic method requires the user to specify a minimum duration for an acceptable pause. Therefore, silent portions in the speech signal that may be pauses but do not exceed this criterion are not investigated (Campbell & Dollaghan, 1995; Green, Beukelman, & Ball, 2004; Walker, Archibald, Cherniak, & Fish, 1992; Yunusova, Weismer, Kent, & Rusche, 2005).

この婉曲的な方法とは対照的に,生理学的な探索は直接吸音と空気の流れと一緒に通して終了したイベントを探索する(Wang, Green, Nip, Kent, Kent, & Ullman, 2010) or chestwall movements (Bunton, 2005; Forner & Hixon, 1977; Hammen & Yorkston, 1994; Hixon, Goldman, & Mead, 1973; Hixon, Mead, & Goldman, 1976; Hoit & Hixon, 1987; Hoit, Hixon, Watson, & Morgan, 1990; McFarland, 2001; Mitchell et al., 1996; Winkworth et al., 1995; Winkworth, Davis, Ellis, & Adams, 1994). 生理学的な探索は適切な計装が必要であり,例えば口頭の空気の流れを計測するためのマスクの着用が必要など,参加者に最低限の負荷がかかる.

In contrast to the indirect methods, the physiologic determination directly detects inspiratory and expiratory events through either airflow (Wang, Green, Nip, Kent, Kent, & Ullman, 2010) or chestwall movements (Bunton, 2005; Forner & Hixon, 1977; Hammen & Yorkston, 1994; Hixon, Goldman, & Mead, 1973; Hixon, Mead, & Goldman, 1976; Hoit & Hixon, 1987; Hoit, Hixon, Watson, & Morgan, 1990; McFarland, 2001; Mitchell et al., 1996; Winkworth et al., 1995; Winkworth, Davis, Ellis, & Adams, 1994). Physiologic detection requires adequate instrumentation and may impose at least slight encumbrances on participants, such as the need to wear a face mask for oral airflow measures.

本稿では短文読み上げタスクにおける Brearh Group のより簡単な探索のためのフォローアップである. Brearh Grop の探索の精度は発話タスクにおそらく影響されるため,最低でも自発音声と音声研究の主要なタスクである短文読み上げでの異なる探索方法のパフォーマンスを調査する必要がある. 研究ではこれら2種類の発話タスクが Brearh Group の構造において多少異なるパターンと関連することを示した(Wang, Green, Nip, Kent, & Kent, 2010).

The present study is a follow-up to an earlier investigation of breath group detection in a task of passage reading. Because accuracy of breath group detection may be affected by the speaking task, it is necessary to examine the performance of different methods of detection in at least spontaneous speech and passage reading, which have been primary tasks in the study of speech production. Studies have shown that these two speaking tasks are associated with somewhat different patterns in breath group structure (Wang, Green, Nip, Kent, & Kent, 2010).

Method (メゾッド)¶

Breath group determination (BreathGroupの特定)¶

pneumotachometerから取得した空気力学的なデータと対話音声信号の刺激は Biopac Student Lab 3.6.7 で記録した. 空気の流れの信号は 1000 Hz でサンプリングし,ローパスフィルター( FLP: 500 Hz )にかけた. 合成された空気の流れの信号はあとで実際の吸気位置の視覚的な特定に使用する. 吸気を特定すると空気の流れの痕跡の上方向のピークで表現される(図1). 信号の中の呼気では下方向に変化する. 吸気位置に関する不確かさがあった数少ない場所は,吸気位置の特定の合意に達するため第一,第二著者が空気の流れの跡を調査した.

注釈

Fig.1

収録された発話サンプルの空気工学的な信号を元にしたドットによる特定された吸気の位置の例示(パネル下). 上部のパネルは音響信号の音圧を示している. 矢印は空気の流れによる探索結果を示す.

A demonstration of the locations of inspiration indicated by the dots for the recorded speech sample based on the aerodynamic signal (the lower panel). The upper panel is the corresponding sound pressure of the acoustic signal. The arrows indicate the direction of airfow

Aerodynamics Data from the pneumotachometer and the simultaneous digital audio signal were recorded using Biopac Student Lab 3.6.7. The airflow signal was sampled at 1000 Hz and subsequently low-pass filtered (FLP 0 500 Hz). The resultant airflow signal was later used to visually identify actual inspiratory loci, represented by the upward peak in the airflow trace indicating inspiration (Fig. 1), whereas a downward trend in the signal indicated expiration. On the rare occasions where there was uncertainty about the location of the inspiratory location, the first and the second authors examined the airflow traces in order to reach a consensus agreement on the inspiratory location.

発話サンプルの知覚的な BreathGrroup は Wisconsin Madison 大学の三人の判断者によって主観的に調査された. 判定者は英語母語話者でありどのように発話の吸気ポーズの信号の知覚的な手がかりを使って BreathGroup を特定するのかを訓練されている. BreathGroupの持続時間のための判定者はどのように利用可能な手がかりの基準を元にBreathGroupの位置を決定するのかについて訓練したあと,彼らの発話を行った. 彼らは,他の対話音声サンプルを聴き,吸気の起きた場所をTranscription Sheetにポイントした. 吸気が聞こえなかった時には,判定者は音声知覚を視覚化情報と例えばパルスの持続時間の長さや,F0の下降,そしてフレーズ末の持続時間の長さなど種々の音響的な手がかりの基準に従い吸気ポイントをマークした. 判断者はタスクの教示説明の基本的なセットも提供された(Appendix参照). 加えて,判定者は Breath Group の位置についての彼らの決定に自身があることを確かにするために発話サンプルを聞いた. Breath Groupの決定の手順は以下の通りである.

1. 音声サンプルは吸音位置の決定の判断としては機能していない転記者によって書き起こしされた.
2. 句読点と大文字小文字の区別(代名詞Iと固有名詞は除く)は書き起こしの句読点やその他の視覚情報から Breath Group を判定してしまうのを防ぐため書き起こしからは排除された. 語順を利用しての判断を防ぐため,それぞれの単語はスペース3つで区切った.
3. 発話サンプルは判定者が発話者の順序をランダマイズされた Breath Group の 探索を行うために,ランダム数の表を使用し,準備された.
4. 判定者は発話サンプルを通常のラウドネスで聴き,吸音だと受け取った位置にマークをした. 判定者は吸音位置を吸音が観測できない場合にはかれらの聴覚印象にしたがって最もよく推測することを依頼された. したがって,これらの判断は聞き手が利用できる多次元的な手がかり,例えばパルス持続時間の長さやF0下降,単語末やシラブルの持続時間の長さ,にもどついている可能性がある. 判定者はBreath Group の位置の推定に満足すうるまで,繰り返し,デジタル化された音声サンプルを聞くことが許された.
5. 吸音位置の知覚的な判定は3人の判定者によって可能なペアすべてを比較し,判定の妥当性を検討した. 信頼性の計測は三人の判定者の知覚的な吸気位置決定の総数で吸気の合意数を割ることで定義した.

Perception Breath groups for the speech samples were determined perceptually by three judges at the University of Wisconsin–Madison. The judges were native English speakers trained on how to identify breath groups using known perceptual cues that signal the production of inspiratory pauses. The judges for the determination of breath group were trained to learn how to determine the location of breath groups on the basis of possible cues before performing their tasks. They were asked to listen to other conversation speech samples and to mark the points on their transcription sheets at which inspiration occurred. When the inspiration was not audible, the judges estimated the inhalation point on the basis of auditory-perceptual impression and various acoustic cues, such as longer pause duration, f 0 declination, and longer phrase-final duration, which are fairly reliable indicators of pauses in normal speech and infant vocalization (Nathani & Oller, 2001; Oller & Lynch, 1992). The judges were also provided with a standard set of instructions explaining the task (see the Appendix). In addition, the judges were allowed to listen to the speech samples repeatedly to ensure that they were confident in their determination on the breath group location. The procedures of breath group determination were as follows:

1. The speech samples were orthographically transcribed by a trained transcriptionist who did not serve as a judge in the determination of inspiratory loci.
2. Punctuations and upper- and lowercase distinctions (except for the pronoun I and proper names) were removed from the orthographic transcripts to prevent the judges from analyzing breath groups on the basis of punctuation and related visual cues in the transcript. Three spaces separated each word to prevent the judges from using word order to separate breath groups.
3. The speech samples prepared for the judges for the task of breath group determination were randomized for order of speaker, using a table of random numbers.
4. The judges listened to the speech samples at normal loudness and marked perceived inspiratory loci on the transcripts. The judges were asked to make a best guess of the inhalation location on the basis of their auditoryperceptual impressions when inspirations were not obvious. Therefore, these judgments could be based on multiple cues available to listeners, such as longer pause duration, f0 declination, and longer phrase-final word or syllable duration. The judges were allowed to listen to the digitized speech samples repeatedly until they were satisfied with their determination of the breath group location.
5. The perceptual judgments of inspiratory loci were compared across each possible pairing of the three judges and across all the three judges to gauge the interjudge reliability. Measurement reliability was defined as the number of points that the judges agreed upon an inspiratory location divided by the total number of perceptually determined inspiratory loci by the three judges.

Results (結果)¶

Accuracy (精度)¶

空気の流れ信号から探索された吸音の総数はすべての話者で1,106個である. SPA アルゴリズムによって探索された 150ms 以上のポーズの総数は2,281個であり,これは判定者が彼らの決定をするための潜在的な吸音の総数であると考えられる.

The total number of inspirations determined from the airflow signal for all speakers was 1,106. The number of pauses greater than 150 ms detected by the SPA algorithm was 2,281, which was considered the total number of potential inspirations for judges to make their decisions.

Perception¶

3人の判定者によって知覚的に探索された吸音位置の総数は1,177である. 判定者1,判定者2,判定者3,によってそれぞれ特定された吸音位置の数は1,088,1,094,1,054である. 3人中最低2人の間で常に判定されたもの(例えばJ1J2,J2J3,J3J1)の総数は1,080である. 3人の中で共通して判定されたものの数は979である. 3人の中で判定間で最も高く一致していた2人の確実性は0.92(1080/1177)である. 3人の判定者間の確実性は0.83である(979/1177).

The total number of inspiratory loci determined perceptually by the three judges was 1,177. The number of inspiratory locations determined individually by judge 1 (J1), judge 2 (J2), and judge 3 (J3) was 1,088, 1,094, and 1,054, respectively. The number of consistent judgments between at least two of the three judges (i.e., J1J2, J1J3, J2J3, or J1J2J3), was 1,080. The number of consistent judgments across all three judges was 979. The highest interjudge reliability between two of the three judges was .92 (1,080/1,177). The interjudge reliability across all the three judges was .83 (979/1,177).

1106個の実際の吸音位置を参照するとJ1は1066個正解し,Missは42個, 22個のFalseがあった. J2は1065, 43, 29個 である. J3は1010, 98, 44個 である. 3人中最低2人の同意のあった位置では 1068個の正解と,40個のMiss,12個のFalseがあった. 3人が全員同意した位置に関しては976個の正解と132個のMiss,3個のFalseがある.

Referenced to the 1,106 actual inspiratory loci, J1 correctly identified 1,066, missed 42, and added 22 (false alarm). J2 correctly identified 1,065, missed 43, and added 29. J3 correctly identified 1,010, missed 98, and added 44. The loci that were consistent between at least two of the three judges were 1,068 correctly identified, 40 missed, and 12 added. The loci that were consistent across all three judges were 976 correctly identified, 132 missed, and 3 added.

表1では特に知覚的判断の 精度,Accuracy,D値を示す. 呼気の場所は 平均正解率(TPR) と False (FPR) とで3人の間で約95%受け入れられた. J1は sensitivity, specify, accuracy, D値,共に一番高かった. 3人の判定者の間で同意のとれた決定は, specificity は常に増加したが, false alarm rate は3人の判定者でばらついた. しかし,3人中最低2人の合意がとれた決定では, specificity は 99% 近くになり, sensitivity, accuracy, D値 はすべて最も高い値になった. 総括すると, 自然発話における吸音位置の知覚的判断の最もよい弁別は3人の判定者のうち最低2人以上の合意に基づいたものであった. しかし,図2の 反応作用曲線(ROC)において示すように, 3人の弁別結果はむしろしっかりクラスタ化された.

Table 1 shows the sensitivity, specificity, accuracy, and d′ data for the perceptual judgments. Inspiratory locations were perceived correctly (TPR) about 95% of the time on average, and the false alarm rate (FPR) varied among the three judges. J1 had the highest sensitivity, specificity, accuracy, and d′. When the decision was based on the agreement across all three judges, the specificity was increased substantially, but the sensitivity and accuracy were decreased to 88%. However, when the decision was based on agreement between at least two of the three judges, the specificity was near 99%, and the sensitivity, accuracy, and d′ were all at their highest. Overall, the best discrimination of the perceptual judgment of inspiratory loci in spontaneous speech was based on the consistency between at least two of the three judges. However, as is shown in the receiver operating characteristic (ROC) curve of Fig. 2, the separate results for the three judges are clustered rather tightly.

Judge(s)		Inspiratory location		TPR	FPR	Accuracy	d-prime	beta (ratio)
		Yes	No
JI	Yes	1066	22	0.9621	0.0188	0.972	3.856	1.799
	No	42	1151
J2	Yes	1065	29	0.9612	0.0247	0.968	3.729	1.452
	No	43	1144
J3	Yes	1010	44	0.9116	0.0375	0.938	3.131	1.96
	No	98	1129
2	Yes	1068	12	0.9639	0.0102	0.977	4.116	2.915
	No	40	1161
3	Yes	976	3	0.8809	0.0026	0.941	3.979	25.122
	No	132	1170

注釈

Table1

The sensitivity, specificity, accuracy, and d-prime data of perceptual judgments determined by judge (J1, J2, J3), by the consistency of at least 2 of the 3 judges, and by the consistency of all the 3 judges. True positive rate (TPR) refers to sensitivity, whereas false positive (FPR) refers to 1- spcificity

注釈

Fig. 2

Receiving operator characteristic curve for the perceptual and acoustic methods of breath group determination. Perceptual results are shown for each judge and agreements between two judges (2) and three judges (3). Acoustic results are shown for various thresholds of pause duration

Acoustics (音響)¶

SPA アルゴリズムによって音響的に検出されたポーズの数は

The number of pauses acoustically determined by the SPA algorithm is given in parentheses in the following summary for the five different pause thresholds: 150 ms (2,281), 200 ms (1,864), 250 ms (1,657), 300 ms (1,513), and 350 ms (1,406). Table 2 shows the sensitivity, specificity, accuracy, and d′ data for the SPA algorithm results. Figure 2 shows the ROC for the combined perceptual and acoustic results. The TPR (sensitivity) values of the five different pause thresholds were all above 98%, but the FPR differed greatly among different threshold values, with smaller thresholds resulting in greater FPRs. The smaller thresholds had near perfect sensitivity but very poor specificity and, consequently, lower accuracy. Thus, in terms of the d′ value, the SPA acoustically determined inspiratory loci of 300-ms threshold had the best performance.

As compared with the actual inspiratory locations determined by the aerodynamic signal, the perceptually determined method with the best performance had smaller TPR and FPR but larger accuracy and d′ than did the acoustically determined method for this spontaneous speech task (Table 2). Moreover, the sensitivity values of the five different pause thresholds were all higher than those of perceptual judgments, but the specificity values were much larger and varied widely (Table 2). Consequently, on the basis of accuracy and d′ analysis, the performance of the perceptually based breath determination of breath groups is judged to be better than that of the acoustic method of pause detection.

Discussion (ディスカッション)¶

The present study indicates that (1) the greatest accuracy in the perceptual detection of inspiratory loci was achieved with agreement between two of the three judges; (2) the most accurate pause duration threshold used for the acoustic detection of inspiratory loci was 300 ms; and (3) the perceptual method of breath group determination was more accurate than the acoustically based determination of pause duration.

For the perceptual approach, the criterion of agreement between two of the three judges yielded the highest TPR, accuracy (.977), and d′ (4.116). This approach had approximately 1.75% (40/2,281) false negatives and 0.53% (12/2,281) false positives. Apparently, the more stringent criterion of consistency across all three judges led to an increase of false negatives that was much larger than the decrease of false positives, thereby reducing both accuracy and d′. In contrast, the most accurate approach for detecting inspiratory loci on the basis of listening in a reading task (Wang, Green, Nip, Kent, Kent, & Ullman, 2010) was agreement across all three judges, which achieved an accuracy of .902, a d′ of 4.140, and a small number of both false negatives (approximately 10%) and false positives (0%). The accuracy of the perceptual approach was better for spontaneous speech in the present study than it was for passage reading in the study by Wang, Green, Nip, Kent, Kent, and Ullman (2010). The differences between spontaneous speech and reading are likely explained by differences in breath group structure, as discussed in Wang, Green, Nip, Kent, and Kent (2010). Breath groups had longer durations for spontaneous speech, as compared with reading. In addition, inspiratory pauses for spontaneous speech are more likely to fall in grammatically inappropriate locations, potentially making the inspirations to be more perceptually salient to the judges.

Using acoustic algorithms to identify inspiratory loci, the optimal threshold of pause detection in the present study was 300 ms, which achieved an accuracy of .817 and a d′ value of 2.994. With this threshold, the false negative rate is 0.2% (5/2,281), but the false positive rate is much higher, approximately 18% (412/2,281). Wang, Green, Nip, Kent, Kent, and Ullman (2010) reported that the most accurate pause duration threshold for detecting inspiratory loci in the reading task was 250, which achieved an accuracy of .895, a d′ of 3.561, a zero rate of false negatives, and an approximately 10% rate of false positives. Task effects between reading and spontaneous speech occurred for the acoustic method, much as they did for the perceptual method. The accuracy and d′ values in spontaneous speech were lower than those in reading. Furthermore, the false negative rate and false positive rate in spontaneous speech were both raised when compared with reading. Consequently, the acoustically determined method in spontaneous speech performed more poorly than for reading, which is likely related to the task differences in the breath group structure and perhaps in cognitive-linguistic load.

Because the minimum inter-breath-group pause in reading for healthy speakers is 250 ms (Wang, Green, Nip, Kent, & Kent, 2010), the 150- and 200-ms thresholds produced no false negatives but many false positives, which lowered their accuracy. In contrast, with thresholds above 200 ms, the decrease in the number of false positives was substantially more than the increase of the number of false negatives, which increased the accuracy. Generally speaking, the false positive rate differed among different pause thresholds, indicating that the selection of the pause threshold is very sensitive to the detection of false positives in spontaneous speech. Because the spontaneous speech samples in the present study were produced fluently by healthy adults who were familiar with the topics to be addressed, there was negligible occurrence of prolonged cognitive hesitations or articulatory or speech errors. Therefore, the present findings may not apply to speech produced by talkers with neurological or other impairments, whose speech might be characterized by either a faster or a slower speaking rate and with more pauses of long durations unrelated to inspiration. A threshold of 300 ms might potentially be either too short for individuals who speak significantly slower or too long for speakers with faster than typical speaking rates.

Taking together the present results and those of Wang, Green, Nip, Kent, Kent, and Ullman (2010), it can be concluded that for both spontaneous speech and passage reading, the perceptual method of breath group determination is more accurate than the acoustic method based on pause duration. The ability of listeners to identify breath groups is no doubt aided by their knowledge that speech is typically produced on a prolonged expiratory phase. Simple acoustic measurements of pauses are naive to this expectation, which is one reason perceptual assessment can be more accurate than acoustic pause detection. The larger d′ obtained for the perceptual approach may indicate that listeners are sensitive to many cues beyond pause duration. Factors related to physiologic needs, cognitive demands, and linguistic accommodations that affect the locations of inspirations and the durations of interbreath-group pauses are possibly perceptible by human ears. Perceptual cues for inspiration include the occurrence of pauses at a major constituent boundary, anacrusis, final syllable lengthening, and final syllable pitch movement (Wozniak et al., 1999). Some of these factors could be included in an elaborated acoustic method that relies on more than just pause duration.

The choice of method for breath group determination should be based on a consideration of the risk–benefit ratio. If errors cannot be tolerated, physiologic methods are preferred, if not mandatory. But if this is not possible (as in the analysis of archived audio signals), the choice between perceptual and acoustic methods should weigh the risk of greater errors (likely to occur with the acoustic method) against the relative costs (in terms of both analysis time and technology). As is shown in Fig. 2, the results for any one judge in the perceptual method were more accurate than those for any of the pause duration thresholds used in the acoustic study. Perceptual determination appears to be a better choice, on the basis of accuracy alone. Of course, these findings pertain to studies interested in identifying only inspiratory pauses, and not those located at phrase and word boundaries; the high false positive rates obtained for the acoustic method suggest that this approach may be well suited for this purpose, although additional research is needed. If it is desired to examine the relationship between breath groups and linguistic structures, preparation of a transcript is necessary for any method of breath group determination. Finally, it should be recognized that the present results and those of Wang, Green, Nip, Kent, Kent, and Ullman (2010) pertain to healthy adult speakers. Generalization of the results to younger or older speakers or to speakers with disorders should be done with caution.

Appendix¶

The instruction of breath group determination for conversational speech samples You will be provided with a transcription of the conversational speech samples without punctuations for each speaker in the present study. The task is to mark the points at which speakers stop for a breath. When you identify this point, place a mark on the corresponding location on the transcript. Make your best guess as to where the speaker stops to take a breath. Sometimes you can hear an expiration and/or inspiration, but in other cases you may have to make the judgment based on other cues, such as longer pause duration, f0 declination, and longer phrasefinal duration. In this task, you can listen to the sound files repeatedly before you are confident in your determination on the breath group location. Do you have any questions?

References¶

• Ainsworth, W. (1973).

  • A system for converting English text into speech.
  • IEEE Transactions on Audio and Electroacoustics, 21, 288–290.

• Bunton, K. (2005).

  • Patterns of lung volume use during an extemporaneous speech task in persons with Parkinson disease.
  • Journal of Communication Disorders, 38, 331–348.

• Bunton, K., Kent, R. D., & Rosenbek, J. C. (2000).

  • Perceptuo-acoustic assessment of prosodic impairment in dysarthria.
  • Clinical Linguistics and Phonetics, 14, 13–24.

• Campbell, T. F., & Dollaghan, C. A. (1995).

  • Speaking rate, articulatory speed, and linguistic processing in children and adolescents with severe traumatic brain injury.
  • Journal of Speech and Hearing Research, 38, 864–875.

• Che, W. C., Wang, Y. T., Lu, H. J., & Green, J. R. (2011).

  • Respiratory changes during reading in Mandarin-speaking adolescents with prelingual hearing impairment.
  • Folia Phoniatrica et Logopaedica, 63, 275–280.

• Collyer, S., & Davis, P. J. (2006).

  • Effect of facemask use on respiratory patterns of women in speech and singing.
  • Journal of Speech Language and Hearing Research, 49, 412–423.

• Forner, L. L., & Hixon, T. J. (1977).

  • Respiratory kinematics in profoundly hearing-impaired speakers.
  • Journal of Speech and Hearing Research, 20, 373–408.

• Green, J. R., Beukelman, D. R., & Ball, L. J. (2004).

  • Algorithmic estimation of pauses in extended speech samples of dysarthric and typical speech.
  • Journal of Medical Speech-Language Pathology, 12, 149–154.

• Hammen, V. L., & Yorkston, K. M. (1994).

  • Respiratory patterning and variability in dysarthric speech.
  • Journal of Medical SpeechLanguage Pathology, 2, 253–261.

• Hixon, T. J., Goldman, M. D., & Mead, J. (1973).

  • Kinematics of the chest wall during speech production: Volume displacements of the rib cage, abdomen, and lung.
  • Journal of Speech and Hearing Research, 16, 78–115.

• Hixon, T. J., Mead, J., & Goldman, M. D. (1976).

  • Dynamics of the chest wall during speech production: Function of the thorax, rib cage, diaphragm, and abdomen.
  • Journal of Speech and Hearing Research, 19, 297–356.

• Hoit, J. D., & Hixon, T. J. (1987).

  • Age and speech breathing.
  • Journal of Speech and Hearing Research, 30, 351–366.

• Hoit, J. D., Hixon, T. J., Watson, P. J., & Morgan, W. J. (1990).

  • Speech breathing in children and adolescents.
  • Journal of Speech and Hearing Research, 33, 51–69.

• Huber, J. E., & Darling, M. (2011).

  • Effect of Parkinson’s disease on the production of structured and unstructured speaking tasks: Respiratory physiologic and linguistic considerations.
  • Journal of Speech, Language, and Hearing Research, 54, 33–46.

• Loudon, R. G., Lee, L., & Holcomb, B. J. (1988).

  • Volumes and breathing patterns during speech in healthy and asthmatic subjects.
  • Journal of Speech and Hearing Research, 31, 219–227.

• MacLarnon, A. M., & Hewitt, G. P. (1999).

  • The evolution of human speech: The role of enhanced breathing control.
  • American Journal of Physical Anthropology, 109, 341–363.

• Macmillan, N. A., & Creelman, C. D. (1991).

  • Detection theory: A user’s guide.
  • New York: Cambridge University Press.

• McFarland, D. H. (2001).

  • Respiratory markers of conversational interaction.
  • Journal of Speech Language and Hearing Research, 44, 128–143.

• Mitchell, H. L., Hoit, J. D., & Watson, P. J. (1996).

  • Cognitive-linguistic demands and speech breathing.
  • Journal of Speech and Hearing Research, 39, 93–104.

• Nathani, S., & Oller, D. K. (2001).

  • Beyond ba-ba and gu-gu: Challenges and strategies in coding infant vocalizations.
  • Behavior Research Methods, Instruments,& Computers, 33, 321–330.

• Oller, D. K., & Lynch, M. P. (1992).

  • Infant vocalizations and innovations in infraphonology: Toward a broader theory of development and disorders.
  • In C. Ferguson, L. Menn, & C. Stoel-Gammon (Eds.), Phonological development: Models, research, implications (pp. 509–536).
  • Parkton, MD: York Press.
  • Behav Res (2012) 44:1121–1128

• Oller, D. K., & Smith, B. L. (1977).

  • Effect of final-syllable position on vowel duration in infant babbling.
  • Journal of the Acoustical Society of America, 62, 994–997.

• Rieger, J. M. (2003).

  • The effect of automatic speech recognition systems on speaking workload and task efficiency.
  • Disability and Rehabilitation, 25, 224–235.

• Schlenck, K. J., Bettrich, R., & Willmes, K. (1993).

  • Aspects of disturbed prosody in dysarthria.
  • Clinical Linguistics & Phonetics, 7, 119–128.

• Walker, J. F., Archibald, L. M., Cherniak, S. R., & Fish, V. G. (1992).

  • Articulation rate in 3- and 5-year-old children.
  • Journal of Speech & Hearing Research, 35, 4–13.

• Wang, Y.-T., Green, J. R., Nip, I. S. B., Kent, R. D., & Kent, J. F.(2010).

  • Breath group analysis for reading and spontaneous speech in healthy adults.
  • Folia Phoniatrica et Logopaedica, 62, 297–302.

• Wang, Y.-T., Green, J. R., Nip, I. S. B., Kent, R. D., Kent, J. F., & Ullman, C. (2010).

  • Accuracy of perceptually based and acoustically based inspiratory loci in reading.
  • Behavior Research Methods, 42, 791–797.

• Wang, Y.-T., Kent, R. D., Duffy, J. R., & Thomas, J. E. (2005).

  • Dysarthria in traumatic brain injury: A breath group and intonational analysis.
  • Folia Phoniatrica et Logopedica, 57, 59–89.

• Winkworth, A. L., Davis, P. J., Adams, R. D., & Ellis, E. (1995).

  • Breathing patterns during spontaneous speech.
  • Journal of Speech and Hearing Research, 38, 124–144.

• Winkworth, A. L., Davis, P. J., Ellis, E., & Adams, R. D. (1994).

  • Variability and consistency in speech breathing during reading: Lung volumes, speech intensity, and linguistic factors.
  • Journal of Speech and Hearing Research, 37, 535–556.

• Wozniak, R. J., Coelho, C. A., Duffy, R. J., & Liles, B. Z. (1999).

  • Intonation unit analysis of conversational discourse in closed head injury.
  • Brain Injury, 13, 191–203.

• Yorkston, K. (1996).

  • Treatment efficacy: Dysarthria.
  • Journal of Speech and Hearing Research, 39, 546–557.

• Yunusova, Y., Weismer, G., Kent, R. D., & Rusche, N. M. (2005).

  • Breath-group intelligibility in dysarthria: Characteristics and underlying correlates.
  • Journal of Speech, Language, and Hearing Research, 48, 1294–1310.

Introduction¶

Recent models of phonetic category acquisition have taken a Mixture of Gaussians approach, in which Gaussian categories are chosen to best fit the distribution of speech sounds in the input, where the input typically consists of isolated speech sounds [1] [2] [3]. One limitation of these approaches is that they do not take into account predictable phonetic variability that is based on context. In actual speech, the acoustic realizations of a phoneme depend on the identities of neighboring sounds, due to coarticulation with these sounds (e.g., as demonstrated in [4] ). This presents a challenge for models of phonetic category acquisition that assume context-independent Gaussian distributions of sounds.

最近の音素カテゴリ獲得のモデルはガウス混合分布アプローチが取られている. このモデルにおいては分離された発話音声からなる典型的なインプットの発話音声の分布に最もよくフィットするガウシアンのカテゴリが選択される. これらのアプローチの限界は,コンテクストに基づく音声的な分散を考慮することができないことである. 実際の発話においては, コーティキュレーションがあるため, 音響的な音素の実体は近隣の音の特性に依存する. 本研究ではコンテクストに独立した音声のガウス分布を仮定した音素カテゴリ獲得モデルの変化を示す.

A more realistic model of phonetic category acquisition should take account of dependencies between neighboring acoustic values that arise due to factors such as gestural overlap and articulatory ease. Despite extensive research on the conditions under which people and animals compensate for coarticulation in speech perception, the problem has only recently begun to be addressed in the context of phonetic category acquisition. There has been a first attempt at solving this problem [5] , in which it is assumed that a learner needs to figure out the direction and magnitude of the shift in acoustic values that occurs in a particular context. This is done by taking the mean of all sound that occur in that context, and comparing it to the mean of all sounds that occur in other contexts.

音素カテゴリ獲得のより現実的なモデルは身振りの重複や音調的容易さのような要素のために起きる近隣の音響的値間の依存度を考慮するべきである. 人間や動物の音声知覚におけるコーアティキュレーションの補償における状態に関する広範囲な研究にもかかわらず, 音素カテゴリの獲得のコンテキストの特定を開始されたのは最近である. この問題の解決する最初の試みは存在し [5] , この研究では学習者が特定のコンテクストにおいて生じる音響的値の遷移の方向と大きさを理解する必要があると仮定している. これは, このコンテキストに生じたすべての音声の平均値を取得し,他のコンテキストで生じるすべての音声の平均とその音声を比較することで行った.

The authors demonstrate that correcting for this shift in the acoustic values makes it easier for a learner to recover a set of Gaussian phonetic categories from acoustic data. However, there is a circularity here, as categories must be known in order to pick out a particular phonological context that would cause an acoustic shift. Thus, it would be desirable for an algorithm to learn both layers of structure simultaneously. As a first step, this paper explores the category learning problem in a system where coarticulatory influences are present, but where the parametric form of these coarticulatory influences is assumed to be known.

作者はこの遷移の音響的値が学習者が音響データからガウシアンの音素カテゴリの再現することを容易にすることで, 正しさを示している. しかし, 音響的遷移を引き起こす特定の音韻的なコンテキストをピックアウトするためにカテゴリを知らなくてはいけないため, ここには矛盾が存在する. したがって, 両方のレイヤーを同時に学習するためのアルゴリズムが望ましい. 最初のステップとして, 本稿ではコーアティキュレーションの影響が存在し,かつ,その値のパラメータが既知であると仮定されるシステムのカテゴリ学習問題を調査した.

Phonological Constraints in Exponential Family Models¶

Phonological constraints were first proposed to be characterized by weighted harmony functions in Harmonic Grammar [6] . Phonetic productions of words in this framework are selected to best satisfy a set of weighted constraints, given the underlying phonological properties of those words. Constraints relate underlying properties to phonetic surface properties (e.g., a constraint would assign higher probability to a /p/ pronounced as [p], as opposed to [b]) and favor certain surface properties over others (e.g., a higher probability is assigned to CV syllables than to CCV syllables). This was later put into a maximum entropy learning framework [7] , where weights are learned for each constraint to maximize the likelihood of a set of training data. In this work, each function \(\phi\) (x) corresponds to a count of the number of times a particular constraint had been violated.

音素カテゴリは Harmonic Grammar [6] において, 重み付けられた調和関数によって決定するために最初に提案された. このフレームワークにおいて,単語の音声学的発声は最も妥当な重み付けられた拘束のセットを選択し,それらの単語の

More recently, the same sort of constraint-based framework has been suggested to be useful for characterizing gradient phonetic effects. Flemming [8] proposed constraints that favor acoustic values similar to a given target production, and also favor similar acoustic values in neighboring speech sounds. Specifically, a speaker is assumed to be minimizing the weighted squared error terms

where each \(\theta\) term is the corresponding weight for a particular constraint, the w terms are the acoustic realizations of neighboring phonemes, and the \(\mu\) terms are targets for specific phonological categories.

This weighted sum squared error cost function corresponds to a particular type of pairwise undirected graphical model (Figure 1). In particular, if the squared error cost functions correspond to the log likelihood of a particular set of values for \(\omega_1\) and \(\omega_2\) , then the potential functions between each 1 pair of nodes are simply Gaussians with variance \(\frac{1}{2\theta}\).

Figure 1: The graphical model corresponding to the model used in [8] .

This problem can be generalized to a series of \(N\) acoustic values, where local potentials are Gaussian around the corresponding category mean \(\mu_c\) with variance \(\Sigma_C\) specific to that category, all pairwise potentials between neighboring acoustic values are Gaussian with common variance \(\Sigma_S\) . The conditional probability \(p(w|z)\) can be expressed as

This can equivalently be expressed in the information form

where each diagonal entry in the matrix \(J\) is \(\sum_{zi}^{-1} + 2\sum_{S}^{-1}\) , except the first and last diagonal entries which are \(\sum_{zi}^{-1} +\sum_{S}^{-1}\) , and the off-diagonal entries are \(−\sum_{S}^{-1}\) for neighboring sounds, zero otherwise.

Each entry in the vector \(h\) is equal to \(\sum_{zi}^{-1}\mu_{zi}\)

The acoustic values \(w_i\) are jointly Gaussian and their marginals can be computed straightforwardly using Gaussian belief propagation. Given category assignments and category parameters, the factor graph is simply a chain with local node potentials and pairwise potentials. The messages originate at the ends of the chain at the variable nodes. Messages from variable nodes to factor nodes are

where the chain structure ensures that there is at most one incoming message to take into account. Messages from factor nodes to variable nodes take the form of

Marginals on node i can be computed straightforwardly as

Learning Model¶

Flemming’s model takes the perspective of a speaker producing an utterance, and assumes that speakers select values of w conditioned on the category assignments \(z\). The work does not address the problems of perception or learning. In the perception and learning problems, a listener or a learner would observe w and need to recover the categories \(z\). Whereas a listener would decide between a set of categories with known means and variances, a learner would need to decide how many categories are in their language, learn the category parameters, and assign each sound to its appropriate category.

Without coarticulation, the learning problem can be characterized as an infinite mixture model (Figure 2a). The learner observes acoustic values and needs to recover the set of categories that generated these acoustic values, and decide which sound belongs to which category. Samples from this posterior distribution can be obtained straightforwardly through Gibbs sampling, as described in [3] .

With coarticulation, the non-parametric model used by [3] can be combined with the framework for weighted constraints used by [8] . A graphical model with the necessary properties is shown in Figure 2b. The key difference in this new model is that the probability distribution \(p(w|z)\) no longer factorizes as \(\prod_i p(w_i|z_i)\). Instead, each zi is generated independently from the Dirichlet process, but the entire acoustic vector w is assumed to be sampled jointly conditioned on all the values of \(z_i\) .

The specific distributions associated with the Dirichlet process are

and the conditional probability distribution \(p(w|z)\) is given by Equation 4.

A learner observes the acoustic values and needs to recover the set of categories that generated the data and decide which sound belongs to which category. Whereas inference in the original model could be done using a collapsed Gibbs sampler, integrating out the category parameters \(\mu_c\) and \(\sigma_c\), integrating over category parameters precludes closed-form calculation of the likelihood function \(p(w|z)\) in the new model because the local potential functions are no longer Gaussian. The sampling algorithm used here therefore explicitly samples the parameters \(\mu_c\) and \(\sigma_c\) for each category.

In each iteration, each category \(z_i\) in turn is chosen conditioned on all other current category assignments and all acoustic values. This conditional probability distribution is

The prior \(p(zi \mid z_{−i} )\) is proportional to the number of sounds already assigned to a particular category, with probability \(\alpha\) of assignment to a new category. The likelihood term \(p(\omega\mid z, \mu, \sigma)\) can be factored as \(p(w_i |z, \mu, \sigma)p(w_{−i} \mid w_i , z_{−i} , \mu, \sigma)\), using the fact that \(w_{−i}\) is independent of \(z_i\) when conditioned on \(w_i\) . The second term does not depend on \(z_i\) and can be ignored. The first term is the marginal probability of \(w_i\) given current category assignments, and can be computed through the message passing algorithm summarized in the previous section. Note, however, that it is not possible to ignore the contribution of \(z_{−i}\) when computing the likelihood term for \(z_i\), as this likelihood term is not conditioned on the values of \(w_{−i}\).

Figure 2: (a) A graphical model of the infinite mixture model. (b) Adapting the model to take into account coarticulation between neighboring sounds.

To estimate the likelihood of a new category, 10 sets of category parameters are sampled from the \(\alpha\) . The likelihood can prior distribution on \(\mu_c\) and \(\sigma_c\), and these are each assigned pseudocounts of 10 be computed for each of these tables as though it were an existing category. If the \(z_i\) being sampled is the only instance of a particular category currently assigned in the corpus, then the parameters from that category are used in place of one of the 10 samples from the prior [9].

The parameters \(\mu_c\) and \(\sigma_c\) should then be resampled for each category, but I could not figure out how to do this, because of the dependencies between different acoustic values. I suspect there is a way to find the posterior on \(\mu_c\) with fixed \(\sigma_c\), which I could not figure out within the time frame for this project, but I’m not sure there is a straightforward way to find the posterior on \(\sigma_c\) at all. In the simulations below I did not ever resample the parameters for existing categories, and any new parameters had to be selected by creating a new category.

Simulations¶

Simulations compared the coarticulation model to the infinite mixture model. For both models, the concentration parameter \(alpha\) was set at 0.1, and the prior over phonetic categories had parameters \(\mu_0 = 0, \sigma_0 = 0.1, and \nu_0 = 0.1\). The coarticulation model was given a fixed parameter of \(\sigma_S = 0.05\). The samplers were each run for 2,000 iterations. For each simulation, pairwise accuracy and completeness measures were used to compute an overall F-score.

Simulation 1 was conducted to demonstrate that the model can take coarticulatory influences into account, finding the correct categories like a regular infinite mixture model but with greater accuracy in category parameters. This is a non-trivial accomplishment because the category parameters are never resampled in the coarticulation model, whereas they are resampled in the infinite mixture model.

One hundred datapoints were generated from two categories with means at -1 and 1 and variances of 0.01. Each category had a mixing probability of \(\frac{1}{2}\) . \(\sigma_S\) was set at 0.05. The resulting corpus is shown in Figure 3.

Both models recovered the two categories perfectly, and assigned sounds to their respective categories correctly, achieving an F-score of 1. However, the learned category parameters were more accurate in the coarticulation model, where the parameters were \(\mu = 1.05, \sigma = 0.01 and \mu = −0.98, \sigma = 0.01\). The infinite mixture model did substantially worse, recovering \(\mu = 0.72, \sigma = 0.03 and \mu = −0.69, \sigma = 0.02\).

Figure 3: Corpus used for Simulation 1. Connected black dots represent the speech sounds in the corpus, and red dots show the means of the categories that generated the sounds.

Figure 4: Corpus used for Simulation 2. Connected black dots represent the speech sounds in the corpus, and red dots show the means of the categories that generated the sounds.

Simulation 2 was a more difficult problem, for which the infinite mixture model could not distinguish two categories. This corpus was created using the same parameters as the previous corpus, but the categories in this corpus had variance 0.5, equal to the coarticulatory variance. This corpus is shown in Figure 4.

The infinite mixture model failed to separate the two categories, instead assigning all sounds to a single category with \(\mu_c = −0.03, \sigma_c = 0.34\). This corresponded to an F-score of 0.66. The coarticulation model also found an incorrect number of categories, separating the points into three categories, corresponding to parameters \(\mu_c = −0.91, \sigma_c = 0.06, \mu_c = 0.97, \sigma_c = 0.04, and \mu_c = −0.04, \sigma_c = 0.001\). However, this last category contained only 7 of the 100 sounds in the corpus. This solution had an F-score of 0.93.

Finally, Simulation 3 tested the models on a more complex corpus created from five categories with more substantial overlap. Sounds were generated from categories with means at -2, -1, 0, 1, and 2. Variances were all set to 0.01, like in Simulation 1. However, because the categories were closer together, and because there were more of them, this was a more difficult learning problem. The corpus is shown in Figure 5. The infinite mixture model merged all five categories into a single categories with \(\mu_c = −0.04, \sigma_c = 1.06\), achieving an F-score of 0.33. The coarticulation model found six categories, with the following parameters:

The last category appears spurious, and indeed contained only 3 sounds from the corpus. This solution had an F-score of 0.89.

Figure 5: Corpus used for Simulation 3. Connected black dots represent the speech sounds in the corpus, and red dots show the means of the categories that generated the sounds.

Discussion¶

This project explored the phonetic category learning problem in a system with coarticulation, where sounds are affected by neighboring sounds. Despite a poor inference algorithm in which no category parameters were resampled for existing categories, the coarticulation model showed the ability to recover parameters of phonetic categories in a toy corpus, recovering more accurate parameters than the infinite mixture model and separating categories that were merged by the infinite mixture model.

Immediate future work should address inference of category parameters, finding a way to recover the posterior distribution on µc and \(\sigma_c\) without the need to invert an NxN matrix. It is likely possible to find the posterior distribution on \(\mu_c\) given a fixed value of \(\sigma_c\) , as the degree of contribution of neighboring sounds changes only with changes in sigma_c .

Language learners need to solve several problems at once: learning category assignments and parameters, but also learning the particular coarticulatory patterns of their language, and sequential dependencies between categories. Dependencies between neighboring categories have been especially difficult to deal with because most work in phonetic category acquisition has used exchangeable models, which by definition assign equal probability to any ordering of sounds in the corpus. This paper has proposed a framework for sequential dependences in which these dependencies are characterized by interacting weighted constraints, following work in formal linguistics [6, 8], and has begun exploring the type of inference that can be performed in such a model.

References¶

Abstract¶

本研究はスペースの空いている本を読んでいるときと,スペースの区切りのない中国語のテキストを読んでいるときの子供と大人の目の動きの行動を調査した. その結果,単語間のスペースは子供も大人も初回のリーディングタイムを減少させ,再固定確率が単語識別と促進された単語の間のスペースを示している. 単語の The present study examined children and adults’ eye movement behavior when reading word spaced and unspaced Chinese text. The results showed that interword spacing reduced children and adults’ first pass reading times and refixation probabilities indicating spaces between words facilitated word identification. Word spacing effects occurred to a similar degree for both children and adults, though there were differential landing position effects for single and multiple fixation situations in both groups; clear preferred viewing location effects occurred for single fixations, whereas landing positions were closer to word beginnings, and further into the word for adults than children for multiple fixation situations. Furthermore, adults targeted refixations contingent on initial landing positions to a greater degree than did children. Overall, the results indicate that some aspects of children’s eye movements during reading show similar levels of maturity to adults, while others do not.

Abstract¶

Before the end of the first year of life, infants begin to lose the ability to perceive distinctions between sounds that are not phonemic in their native language. It is typically assumed that this developmental change reflects the construction of language-specific phoneme categories, but how these categories are learned largely remains a mystery. Peperkamp, Le Calvez, Nadal, and Dupoux (2006) present an algorithm that can discover phonemes using the distributions of allophones as well as the phonetic properties of the allophones and their contexts. We show that a third type of information source, the occurrence of pairs of minimally differing word forms in speech heard by the infant, is also useful for learning phonemic categories and is in fact more reliable than purely distributional information in data containing a large number of allophones. In our model, learners build an approximation of the lexicon consisting of the high-frequency n-grams present in their speech input, allowing them to take advantage of top-down lexical information without needing to learn words. This may explain how infants have already begun to exhibit sensitivity to phonemic categories before they have a large receptive lexicon.

• Keywords: First language acquisition; Statistical learning; Phonemes; Allophonic rules

1.Introduction¶

Infants acquire the fundamentals of their language very quickly and without supervision. In particular, during the first year of life, they converge on the set of phonemic categories for their language (Kuhl et al., 2008; Werker & Tees, 1984), they become attuned to its phonotactics (Jusczyk, Friederici, Wessels, Svenkerud, & Jusczyk, 1993), and they begin to extract words from continuous speech (Jusczyk & Aslin, 1995). Despite a wealth of studies documenting these changes (see reviews in Jusczyk, 2000; Kuhl, 2004; Werker & Tees, 1999), very little is known about the computational mechanisms involved in this rapid acquisition process.

• Correspondence should be sent to Andrew Martin, Laboratory for Language Development, RIKEN Brain Science Institute, 2-1 Hirosawa, Wako-shi, Saitama, 351-0198, Japan. E-mail: amartin@brain.riken.jp

One reason for this state of affairs is that most computational modeling studies have narrowly focused on one learning sub-problem, assuming the others can be solved independently. For instance, proposed mechanisms for word segmentation have typically assumed that phonemic categories have been acquired beforehand (Brent, 1999; Goldwater, Griffiths, & Johnson, 2009; Venkataraman, 2001). Similarly, models of the phonetic ⁄ phonological buildup of phonemic categories assume that these categories have to be constructed or refined through generalization over lexical items (Pierrehumbert, 2003). Of course, this presupposes that a lexicon has been acquired beforehand. Finally, models of grammar induction for learning stress systems or phonotactics make the assumption that both the phonemic categories and the lexicon of underlying forms have already been learned (Dresher & Kaye, 1990; Tesar & Smolensky, 1998). These circularities illustrate what is known as the ‘‘bootstrapping problem,’’ which is the apparently insoluble problem that faces infants when they have to acquire several co-dependent levels of linguistic structure. Therefore, while the existing computational studies have allowed us to better understand individual pieces of the puzzle, taken together, they do not coalesce into a coherent theory of early language acquisition. This is especially true when one considers that the experimental data show that infants do not learn phonemic categories, lexical entries, and phonotactic regularities one after the other, but rather, start learning these levels almost simultaneously, between 5 and 9 months of age (see Kuhl, 2004 for a review).

A second reason for our lack of comprehension of this learning process is that most proposed algorithms have only been tested on artificial or simplified language inputs, and only assume that they can scale up to more realistic inputs like raw speech; in several instances, however, such assumptions have turned out to be incorrect. For example, it has been claimed that phonetic categories emerge through an unsupervised statistical learning process whereby infants track the modes of the distributions of phonetic exemplars (Maye, Weiss, & Aslin, 2008; Maye, Werker, & Gerken, 2002) or perform perceptual warping of irrelevant phonetic dimensions (Jusczyk, 1993; Kuhl et al., 2008). Modeling studies have found that Self Organizing Maps (Gauthier, Shi, & Xu, 2007a,b; Guenther & Gjaja, 1996), Gaussian mixtures (de Boer & Kuhl, 2003; Mcmurray, Aslin, & Toscano, 2009; Vallabha, McClelland, Pons, Werker, & Amano, 2007), or Hidden Markov Models (Goldsmith & Xanthos, 2009) can converge on abstract categories for tones, vowels, or consonants in an unsupervised fashion, that is, without any kind of lexical or semantic feedback. However, these clustering algorithms have only been applied to a fragment of the phonology or to individual speech dimensions segmented or extracted by hand (such as F0 contours, F1 and F2, vowel duration, or VOT). Do they scale up to the full complexity of unsegmented speech signals? Varadarajan, Khudanpur, and Dupoux (2008) applied unsupervised clustering (a version of Gaussian modeling using successive state splitting) on 40 h of raw conversational speech and found that the algorithm constructed a network of states that failed to correspond in a one-to-one fashion to phonemic categories. Although the states were sufficient to sustain state-of-the-art speech recognition performance, they did so by encoding a very large number of heavily context-dependant discrete acoustic (subphonemic) events. Evidently, blind unsupervised clustering on raw speech needs to be supplemented by further processes in order to adequately model the early construction of phonemic categories by infants.

It is important to realize that the failure to derive abstract phonemic categories from the raw signal using unsupervised clustering in Varadarajan et al. (2008) is not a fluke of the particular algorithm they used but reflects a deep property of running speech signals: The acoustic realization of phonemes is massively context-dependent, creating a great deal of overlap between phoneme categories (see Pierrehumbert, 2003). One could argue that gradient coarticulation effects and abstract linguistic rules are fundamentally different, in that the former are the result of universal principles which could be undone by infants without needing any language-specific knowledge. If this were the case, perhaps the wealth of variants discovered by ASR systems could be reduced to a small, orderly set of allophones. There is substantial evidence, however, that even fine-grained coarticulation varies across languages ¨ (Beddor, Harnsberger, & Lindemann, 2002; Choi & Keating, 1991; Manuel, 1999; Ohman, 1966), and that perceptual compensation on the part of listeners shows language-specific effects (Fowler, 1981; Fowler & Smith, 1986; Lehiste & Shockey, 1972). The boundary between gradient phonetic effects and discrete phonological rules is thus difficult to draw, especially from the viewpoint of infants who have not yet acquired a system of discrete categories.

Constructing a single context-independent Gaussian model for each abstract phoneme is therefore bound to yield more identification errors when compared to finer-grained models that make use of contextual information. This is well known in the speech recognition literature, where HMM models of abstract phonemes yield worse performance than models of contextual diphone- or triphone-based allophones (Lee, 1988; Makhoul & Schwartz, 1995). Typically, a multi-talker recognition system requires between 2,000 and 12,000 contextual allophones (representing between 50 and 300 allophones per phoneme) in order to achieve good performance (Jelinek, 1998).

Of course, one could propose that infants adopt the same strategy—simply compile a large number of such fine-grained allophonic categories and use them for word learning. But this would be unwise, since massive allophony causes the performance of word segmentation algorithms to drop dramatically (Rytting, Brew, & Fosler-Lussier, 2010). Boruta, ´ Peperkamp, Crabbe, and Dupoux (2011) ran two such algorithms—the MBDP-1 of Brent (1999) and the NGS-u of Venkataraman (2001)—on transcripts of child-directed speech in which contextual allophony has been implemented. They found that when the input contains an average of 20 allophones per phoneme, the performance of both algorithms falls below that of a control algorithm which simply inserts word boundaries at random. In addition, there is empirical evidence that infants do not do this, since before they have a large comprehension lexicon toward the end of their first year of life,1 they have begun to lose the ability to discriminate within-category distinctions (e.g., Dehaene-Lambertz & Baillet, 1998; Werker & Tees, 1984) and pay less attention to allophonic contrasts compared to phonemic ones (Seidl, Cristia, Bernard, & Onishi, 2009).

This does not mean that fine-grained phonetic information is necessarily disregarded, as speakers could make use of both phonetic detail and abstract categories (Cutler, Eisner, McQueen, & Norris, 2010; Ramus et al., 2010). Evidence that phonetic detail is represented includes results showing that both infants and adults use allophonic differences for the purposes of word segmentation (Gow & Gordon, 1995; Jusczyk, Hohne, & Bauman, 1999; Nakatani & Dukes, 1977), and that adults are able to remap allophones to abstract phonemes with little training (Kraljik & Samuel, 2005; Norris, McQueen, & Cutler, 2003). Similarly, both adults and infants make use of within-category phonetic variation during lexical access (McMurray & Aslin, 2005; McMurray, Aslin, Tanenhaus, Spivey, & Subik, 2008).

In this paper, we revisit the issue of early acquisition of phoneme categories, while attempting to address simultaneously the two problems mentioned above, that is, the circularity problem and the scalability problem. We build our approach on the work described in two recent papers: Peperkamp et al. (2006) and Feldman, Griffiths, and Morgan (2009).

Peperkamp et al. (2006) proposed that infants construct phoneme categories in two steps: Starting with raw signal, they first construct a rather large number of detailed phonetic categories (allophones), and in a second step they cluster these allophones into abstract phonemes. They demonstrated that two potential sources of information, distributional and phonetic, are potentially helpful in performing this clustering step. Distributional information is useful for category formation because variants of a single phoneme occur in distinct contexts, while phonetic information is relevant due to the fact that such variants tend to be acoustically or articulatorily similar to each other and to share phonetic features with their contexts. However, the phonetic part of the study may have created a circularity problem since it assumed the prior acquisition of phonetic features (which may itself require learning the phonological system; see Clements, 2009). In addition, the entire study may have a scalability problem, as it only tested a very small number of allophones in each language (7 allophones of French in Peperkamp et al., 2006; 15 allophones of Japanese in Le Calvez, Peperkamp, & Dupoux, 2007). This falls short of the range of allophonic ⁄ coarticulatory variants that are needed to achieve reasonable performance in speech recognition, which are on the order of a few thousand variants. Since we do not know the granularity of the categories constructed by actual infants, in the present study we manipulate the number of variants in a parametric fashion, from a few dozen to a few thousand.

Feldman et al. (2009) showed, using a Bayesian model, that it can be more beneficial to simultaneously learn a lexicon and phoneme categories than to learn phoneme categories alone. While non-intuitive, this result makes an extremely important conceptual point, as it shows that potential circularities between learning at the lexical and sublexical levels can be broken up using appropriate learning algorithms, where bottom-up and top-down information are learned simultaneously and constrain each other (see also Swingley, 2009). However, their study may also rest on a potential circularity problem since the words provided to the model were all segmented by hand (whereas, as we know, segmentation depends on the availability of abstract enough phonemic categories; Boruta et al., 2011). It may also have a scalability problem, as it modeled the acquisition problem with toy examples, consisting of a small number of artificial or phonetically idealized categories and a small number of words. Here, we will expand on Feldman et al.’s (2009) idea of simultaneous learning of phonemic units and words, while using a realistically sized corpus as in Peperkamp et al. (2006), incorporating the full phoneme inventory, phonotactic constraints and lexicon. Importantly, in order to avoid the circularity problem mentioned above, we will use an approximate proto-lexicon derived in an unsupervised way from the sublexical representation, without any kind of hand segmentation.

In brief, we combine ideas from both Peperkamp et al. (2006) and Feldman et al. (2009). Following the former, we assume that some kind of initial clustering has yielded a set of discrete segment candidates (the allophonic set). The model’s task is then to group them into equivalence classes in order to arrive at abstract phonemes (the phonemic set). Our approach is not aimed at producing a realistic simulation or an instantiated theory of phonological acquisition in infants. Rather, we are interested in quantitatively evaluating the usefulness of different kinds of information that are available in infant’s input for the purpose of learning phonological categories.

In Experiment 1 we study the scalability of the bottom-up distributional measure used in Peperkamp et al. (2006) and Le Calvez et al. (2007) when the number of allophones is increased beyond an average of two per phoneme. We also implement two types of allophone: those defined by the following context (Experiments 1 and 2), as in Peperkamp et al. (2006) and Le Calvez et al. (2007), and those conditioned by bilateral contexts (Experiment 3), which mimic the triphones used in ASR and allow us to test an even larger number of allophones per phoneme. In Experiments 2 and 3 we implement a new algorithm, incorporating the idea of Feldman et al. (2009) and Swingley (2009) that feedback from a higher level can help the acquisition of a lower level, even if the higher level has not yet been perfectly learned. To assess this quantitatively, we compare the effect of a perfect (supervised) lexicon with that of an approximate proto-lexicon derived by means of a simple unsupervised segmentation mechanism. We conclude by discussing the predictions of this new model regarding the existence and role of approximate word forms during the first year of life.

2.Experiment 1¶

In this experiment, we examine the performance of Peperkamp et al.’s (2006) algorithm, which uses Kullback–Leibler (KL) divergence as a measure of distance between contexts to detect pairs of allophones that derive from the same phoneme, on corpora with varying numbers of allophones. A pair of segments with high KL divergence (i.e., having dissimilar distributions) is deemed more likely to belong to the same phoneme than one with low KL divergence. The robustness of this algorithm has been demonstrated using pseudo-random artificial corpora, as well as transcribed corpora of French and Japanese infant-directed speech in which nine and fifteen allophones, respectively, had been added to the phoneme inventory (Le Calvez et al., 2007; Peperkamp et al., 2006). It has also been successfully used on the consonants in the TIMIT English database, whose transcriptions include three allophones in addition to the standard phonemic symbols (Kempton & Moore, 2009). In this experiment we greatly increase the number of allophones in the training data in order to determine whether this method can scale up to systems of realistic complexity.

3.Experiment 2¶

The solution we propose to the dilemma posed by high allophonic complexity takes advantage of the fact that the linguistic input is composed of words. A pair of phonological rules which change phoneme x into x1 when followed by y and into x2 when followed by z will cause most words that end in the phoneme x to occur in two variants: one that ends in x1 (when the word is followed by y) and one that ends in x2 (when the word is followed by z). Thus, encountering a pair of word forms that differ only in that one ends in x1 and the other ends in x2 is a clue that x1 and x2 are allophones of the same phoneme—conversely, never encountering such a word form pair (in a sufficiently large sample) is a clue that x1 and x2 are allophones of different phonemes. The Japanese word atama ‘‘head,’’ for example, appears as a1t1a2m2a2 before the nominative marker ga in the utterance in Fig. 1, but it would appear as a1t1a2m2a1 before the word to ‘‘and.’’ The presence of both word forms in the infant’s input is evidence that a1 and a2 are allophones of the phoneme ⁄ a ⁄ . This is not an infallible learning strategy, since every language contains minimal pairs, different words which by chance differ only in a single segment (e.g., kiku ‘‘listen’’ and kiki ‘‘crisis’’ in Japanese), which could result in allophones of different phonemes being misclassified as belonging to the same phoneme. However, as long as the number of minimal pairs is small relative to the number of word form pairs derived from the same word, the strategy will be effective.

A learner who knows where the word boundaries are could filter out those segment pairs that are not responsible for multiple word forms, before using KL divergence to classify the remaining pairs. Of course, such a word form filter would unrealistically require perfect knowledge of word boundaries, and thus the set of attested word forms, on the part of the learner. Infants who have not discovered word boundaries yet, however, can approximate the set of word forms by compiling a list of high-frequency strings that occur in the input. In this experiment, we implement both a word form filter and an n-gram filter; the latter is identical to the former except that the set of strings used to construct it is made up of the most frequent n-grams occurring in the corpus for a range of values of n.

4. Experiment 3¶

Experiments 1 and 2 use phonological rules that are unilaterally conditioned, in particular, rules that are triggered by the phoneme’s following context. Actual phonological processes in natural languages, however, are often conditioned by bilateral contexts. In Korean, for example, stop consonants become voiced when both preceded and followed by a voiced segment (Cho, 1990). In this Experiment we therefore test the algorithms used in Experiments 1 and 2 on data in which allophones are dependent on both the preceding and the following segment.

5. General discussion¶

The development of phonetic perception over the first year of life poses a conundrum. By the end of this period, despite having little access to semantic information, infants treat semantically meaningful (i.e., phonemic) and meaningless (i.e., allophonic) phonetic distinctions differently. We have argued that one way out of this conundrum for infants involves building phonemic categories, effectively classifying these distinctions as either phonemic or allophonic, using a procedure that exploits the lexical structure of the input.

We have shown, firstly, that when searching for phonemic categories, a bottom-up procedure which looks for sounds that are in complementary distribution becomes extremely inefficient when allophonic complexity (i.e., the number of allophones) increases. Secondly, we found that adding top-down lexical word form information allows for robust discrimination among segment pairs that belong to the same phoneme and those that belong to different phonemes, even in the presence of increased allophonic complexity. Finally, we have shown that lexical word forms can be crudely approximated with n-grams, which still yield results that are both good and resistant to allophonic complexity. These results are obtained with the same types of contextual variants as used in Peperkamp et al. (2006) and Le Calvez et al. (2007), that is, allophones that depend upon the following context (Experiments 1 and 2), as well as with bilaterally conditioned allophones (Experiment 3). Moreover, the results hold for both Dutch and Japanese, two languages that are very different from the viewpoint of syllable structure and phonotactics.

The reason for the lack of robustness of the bottom-up complementary distribution approach is fairly simple to grasp: as the number of allophones increases, the allophones become more and more tied to highly specific contexts. Ultimately, in a language where all possible bilateral allophones are instantiated, every segment is in complementary distribution with almost every other one, rendering distributional information nearly useless for phonemic categorization. Only when the number of allophonic variants is very small (not more than two per phoneme), and complementary distribution of segments thereby relatively rare, is this type of distributional information useful. Unfortunately for the learner, this means that looking for complementary distribution between segments in the input is only an efficient strategy when the problem has already been almost completely solved.

The top-down approach is successful because it relies on the fact that allophony changes not just individual sounds, but entire word forms, and that for sufficiently long words, the probability that two different words happen to be identical except for their final sounds is very low. Crucially, this fact is independent of the allophonic complexity of the input. Of course, this criterion alone is not sufficient, as there are true minimal pairs in many languages, but they are relatively rare (especially for longer words) and non-systematic, unlike the minimal pairs created by contextual allophony. Finally, the reason for the success of the n-gram strategy is that the low probability of true minimal pairs also applies to frequent n-grams, and this probability does not depend on allophonic complexity.

Our algorithm could be improved and extended in a number of ways. First, instead of using top-down information in an all-or-none fashion, we could implement a statistical model of possible word forms (Swingley, 2005) and use it to compute the probability of obtaining the observed pattern of minimal pairs by accident. Such a procedure would allow us to probabilistically integrate the effect of word length instead of postulating a fixed window of word lengths as in the present study, and it would also be less sensitive to the occasional existence of true minimal pairs. Second, the crude n-gram segmentation strategy could be replaced by some less crude—although still sub-optimal—lexical segmentation procedure (e.g., Brent, 1999; Daland & Pierrehumbert, 2010; Goldwater et al., 2009; Monaghan & Christiansen, 2010; Venkataraman, 2001). Third, as noted in Peperkamp et al. (2006), performance can be improved by providing phonetic constraints as to the nature of allophonic processes. Peperkamp et al., 2006. proposed two such constraints, one to the effect that allophones of the same phoneme must be minimally different from a phonetic point of view, and another to the effect that allophones tend to result from contextual assimilation (feature spreading). How such phonetic constraints can be implemented in a language encoded with massive allophony remains to be assessed. Another example of a possibly helpful linguistic constraint is rudimentary semantic knowledge, which could serve as further evidence that two word forms are in fact realizations of a single word. Even if infants do not know many words, the few words they do know could improve the performance of the n-gram filter (Swingley, 2009). Fourth, our model could be extended to learn patterns of phonological variations that go beyond contextual allophony. Word forms can also vary through processes of insertion or deletion of segments, yielding a multiplication of closely related word forms (Greenberg, 1998). A proto-lexicon has the potential of capturing some of these variations, which would be impossible to do in a purely bottom-up fashion (Peperkamp & Dupoux, 2002).

Of course, the procedure we have described represents only the first step in the learning of phonemic categories. Our algorithm assigns each pair of allophones a rating which indicates how likely that pair is to belong to the same phoneme; a real infant would need to then use these ratings to group allophones into phoneme-sized clusters of allophones. This next step is not trivial. Given pairs of allophones and ratings, the infant must decide on an appropriate cutoff value above which allophones will be considered members of the same cluster. Choosing the optimal cutoff will depend on the relative costs of false alarms (allophones incorrectly grouped together) and misses (allophones incorrectly placed in different clusters). At this point, very little is known about what these costs might be or how easy it is for infants to recover from errors at this stage of learning. Although modeling this entire process is thus beyond the scope of the present article, we hope that our results provide a foundation for future progress on these questions.

We also emphasize that, although we have couched our proposal in terms of a specific algorithm, our findings allow us to draw more general conclusions that go beyond the question of whether infants use this exact procedure to learn phonemes. These results demonstrate, first, that the lexical structure of speech input contains information on phonemic categories that is missed by approaches which focus only on sublexical units, and second, that this lexical information can be extracted from data of realistic complexity, even using an extremely simple procedure. It is therefore likely that any approach to learning phonemes would benefit from making use of top-down lexical information.

We should, however, point out that our approach employs a number of simplifications that make it unable to address the entire complexity of the acquisition problem. First, the assumption that the learner starts by establishing a large set of discrete allophones may not adequately capture some of the phonetic effects of between-talker variation, nor the more continuous effects of speaking rate, or variability induced by noise in the transmission channel. Clearly, adequate signal preprocessing ⁄ normalization is needed if a running speech application is envisioned. Second, as mentioned earlier, the use of a minimal pair constraint may be problematic in languages with mono-segmental inflectional affixes that create systematic patterns of word-final or word-initial minimal pairs (as in the Japanese aruku-aruke example mentioned in section 3.2). To solve these cases, the proto-lexicon of word forms must be supplemented with semantic information which may only be acquired later during development (Regier, 2003). This is consistent with the view that the acquisition of phonemic categories is not closed by the end of the first year of life but continues to be refined thereafter (Sundara, Polka, & Genesee, 2006). Third, we should note that our procedure can only discover phonological rules that operate at word boundaries; a context-dependent rule that only ever applies within a word will not create word form variants in the way discussed here and will have to be learned in some other way. But precisely because such rules, if applied consistently, do not create multiple word forms, they do not impede word recognition or segmentation and so are not as crucial for a language-learning infant.

The traditional bottom-up scenario of early language acquisition holds that infants begin by learning the phonemes and constraints upon their sequencing during the first year of life (Jusczyk et al., 1993; Pegg & Werker, 1997; Werker & Tees, 1984) and then learn the lexicon on the basis of an established phonological representation (Clark, 1995). While infants have been shown to be capable of extracting phonological regularities in a non-lexical, bottom-up fashion (Chambers, Onishi, & Fisher, 2003; Maye et al., 2002; Saffran & Thiessen, 2003; White, Peperkamp, Kirk, & Morgan, 2008), our results cast serious doubt on the idea that such a mechanism is by itself sufficient to establish phonological categories. Indeed, we have shown that attempts to re-cluster allophones in a bottom-up fashion based on complementary distributions is inefficient in the face of massive allophony. However, we also showed that it is possible to replace the bottom-up scenario with one that is nearly the reverse, in which an approximate lexicon of word forms is used to acquire phonological regularities. In fact, an interactive scenario could be proposed, in which an approximate phonology is used to yield a better lexicon, which in turn is used to improve the phonology, and so on, until both phonology and the lexicon converge on the adult form (Swingley, 2009).

The present approach opens up novel empirical research questions in infant language acquisition. For instance, does a proto-lexicon exist in infants before 12 months of age? If so, what are its size and growth rate? Ngon et al. (in press) provide preliminary answers to these questions. They found that 11-month-old French-learning infants are able to discriminate highly frequent n-grams from low-frequency n-grams, even when neither set of stimuli contained any actual words. This suggests that at this age, infants have indeed constructed a proto-lexicon of high-frequency sequences which consists of a mixture of words and nonwords. The Ngon et al. (in press). study raises the question of how to estimate the size and composition of the proto-lexicon as a function of age. Such an estimation should be linked to modeling studies like the present one, in order to determine the extent to which the protolexicon can help the acquisition of phonological categories.4

There are other questions raised by our results that will be more difficult to answer. In particular, does the growth of the proto-lexicon predict the acquisition of phonological categories and of phonological variation? And how does the acquisition of phonology help the acquisition of a proper lexicon of word forms? These questions have been neglected in the past, perhaps because of the belief that a proper lexicon cannot be learned before phonemic categories are acquired. The present results, however, suggest that an understanding of lexical acquisition will be a fundamental component of a complete theory of phonological acquisition. Clearly, more research is needed to understand the mechanisms that could make it possible to simultaneously learn lexical and phonological regularities, and whether infants can use these mechanisms during the first year of life.

Appendix: Kullback-Leibler measure of dissimilarity between two probability distributions¶

Let s be a segment, c a context, and PðcjsÞ the probability of observing c given s. Then the Kullback-Leibler measure of dissimilarity between the distributions of two segments s1 and s2 is defined as:

数式

with 数式

with n(c,s) the number of occurrences of the segment s in the context c (i.e., the number of occurrences of the sequence sc),

• n(s) the number of occurrences of the segment s,
• and N the total number of contexts.
• In order to smooth the probability estimates of the distributions in finite samples, 1 ⁄ N occurrence of each segment is added in each context, where N is the total number of contexts.

Notes¶

1. Dale and Fenson (1996) found that English-learning 11-month-old infants comprehended an average of 113 words.
2. Note that contexts for our rules are limited to underlying phonemes. In actual languages, the outputs of some rules can serve as the inputs to other rules, further complicating the learning process.
3. A number of words present in the Dutch orthographic corpus (largely proper nouns) were not listed in the pronunciation lexicon. We eliminated any utterances containing these words from our corpora, resulting in a roughly 20% reduction in corpus size.
4. For instance, the present algorithm uses the 10% most frequent n-grams as a protolexicon. Given the size of the corpus, this turns out to be a rather large set (over a million word forms). The use of a more realistic segmentation procedure would certainly cut down this number and bring it closer to the size of the protolexicon as it could be measured in infants.
5. With bilateral contexts, implementing our algorithm becomes computationally prohibitive on the most complex corpora. The rightmost points in Fig. 4 represent the most complex corpora we were able to process given our available resources.

Acknowledgments¶

This research was made possible by support from the Centre National de la Recherche Scientifique and the RIKEN Brain Science Institute, in addition to grants ANR-2010BLAN-1901-1 from the Agence Nationale pour la Recherche and ERC-2011-AdG295810 from the European Research Council.

References¶

• Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12, 387–415.
• Beddor, P. S., Harnsberger, J. D., & Lindemann, S. (2002). Language-specific patterns of vowel-to-vowel coarticulation: acoustic structures and their perceptual correlates. Journal of Phonetics, 30 (4), 591–627.
• de Boer, B., & Kuhl, P. K. (2003). Investigating the role of infant-directed speech with a computer model. Acoustics Research Letters Online, 4(4), 129–134.
• Boruta, L., Peperkamp, S., Crabbe, B., & Dupoux, E. (2011). Testing the robustness of online word segmentation: Effects of linguistic diversity and phonetic variation. Proceedings of CMCL, ACL., 2, 1–9.
• Brent, M. R. (1999). An efficient, probabilistically sound algorithm for segmentation and word discovery. Machine Learning, 34(1–3), 71–105.
• Chambers, K. E., Onishi, K. H., & Fisher, C. (2003). Infants learn phonotactic regularities from brief auditory experience. Cognition, 87(2), B69–B77.
• Cho, Y. Y. (1990). Syntax and phrasing in Korean. In S. Inkelas & D. Zec (Eds.), The phonology-syntax connection (pp. 47–62). Chicago: University of Chicago Press.
• Choi, J. D., & Keating, P. (1991). Vowel-to-vowel coarticulation in three Slavic languages. UCLA Working Papers in Phonetics, 78, 78–86.
• Clark, E. V. (1995). The lexicon in acquisition. Cambridge, England: Cambridge University Press.
• Clements, G. N. (2009). The role of features in phonological inventories. In E. Raimy & C. E. Cairns (Eds.),
• Contemporary views on architecture and representations in phonological theory (pp. 19–68). Cambridge, MA: MIT Press.
• Cutler, A., Eisner, F., McQueen, J., & Norris, D. (2010). How abstract phonemic categories are necessary for coping with speaker-related variation. Papers in Laboratory Phonology, 10, 91–111.
• Daland, R., & Pierrehumbert, J. (2010). Learning diphone-based segmentation. Cognitive Science, 35(1), 119–155.
• Dale, P., & Fenson, L. (1996). Lexical development norms for young children. Behavior Research Methods, 28(1), 125–127.
• Dehaene-Lambertz, G., & Baillet, S. (1998). A phonological representation in the infant brain. NeuroReport, 9(8), 1885.
• Dresher, B. E., & Kaye, J. D. (1990). A computational learning model for metrical phonology. Cognition, 34(2), 137–195.
• Feldman, N., Griffiths, T., & Morgan, J. (2009). Learning phonetic categories by learning a lexicon. Proceedings of the 31st Annual Conference of the Cognitive Science Society, 2208–2213.
• Fowler, C. A. (1981). Production and perception of coarticulation among stressed and unstressed vowels. Journal of Speech and Hearing Research, 24, 127–139.
• Fowler, C. A., & Smith, M. (1986). Speech perception as ‘‘vector analysis’’: An approach to the problems of segmentation and invariance. In J. S. Perkell & D. Klatt (Eds.), Invariance and variability of speech processes (pp. 123–136). Hillsdale, NJ: Lawrence Erlbaum Associates.
• Gauthier, B., Shi, R., & Xu, Y. (2007a). Learning phonetic categories by tracking movements. Cognition, 103 (1), 80–106.
• Gauthier, B., Shi, R., & Xu, Y. (2007b). Simulating the acquisition of lexical tones from continuous dynamic input. Journal of the Acoustical Society of America, 121(5), EL190–EL195.
• Goldsmith, J., & Xanthos, A. (2009). Learning phonological categories. Language, 85(1), 4–38.
• Goldwater, S., Griffiths, T. L., & Johnson, M. (2009). A Bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112, 21–54.
• Gow, D. W. Jr, & Gordon, P. C. (1995). Lexical and prelexical influences on word segmentation: Evidence from priming. Journal of Experimental Psychology: Human Perception and Performance, 21(2), 344–359.
• Greenberg, S. (1998). Speaking in shorthand—A syllable-centric perspective for understanding pronunciation variation. Proceedings of the ESCA Workshop on Modeling Pronunciation Variation for Automatic Speech Recognition, 47–56.
• Guenther, F. H., & Gjaja, M. N. (1996). The perceptual magnet effect as an emergent property of neural map formation. Journal of the Acoustical Society of America, 100(2), 1111–1121.
• Jelinek, F. (1998). Statistical Methods of Speech Recognition. Cambridge, MA: MIT Press.
• Jusczyk, P. W. (1993). From general to language-specific capacities: The WRAPSA model of how speech perception develops. Journal of Phonetics, 21, 3–28.
• Jusczyk, P. (2000). The discovery of spoken language. Cambridge, MA: MIT Press.
• Jusczyk, P. W., & Aslin, R. N. (1995). Infants’ detection of the sound patterns of words in fluent speech. Cognitive Psychology, 29(1), 1–23.
• Jusczyk, P. W., Friederici, A., Wessels, J., Svenkerud, V., & Jusczyk, A. (1993). Infants’ sensitivity to the sound patterns of native language words. Journal of Memory and Language, 32 (3), 402–420.
• Jusczyk, P. W., Hohne, E. A., & Bauman, A. (1999). Infants’ sensitivity to allophonic cues for word segmentation. Perception & Psychophysics, 61(8), 1465.
• Kempton, T., & Moore, R. K. (2009). Finding allophones: An evaluation on consonants in the TIMIT corpus. Interspeech, 2009, 1651–1654.
• Kraljik, T., & Samuel, A. (2005). Perceptual learning in speech: Is there a return to normal? Cognitive Psychology, 51, 141–178.
• Kuhl, P. K. (2004). Early language acquisition: cracking the speech code. Nature Reviews Neuroscience, 5(11), 831–843.
• Kuhl, P. K., Conboy, B. T., Coffey-Corina, S., Padden, D., Rivera-Gaxiola, M., & Nelson, T. (2008). Phonetic learning as a pathway to language: New data and native language magnet theory expanded (NLM-e). Philosophical Transactions of the Royal Society B, 363, 979–1000.
• Le Calvez, R., Peperkamp, S., & Dupoux, E. (2007). Bottom-up learning of phonemes: a computational study. Proceedings of the Second European Cognitive Science Conference, 2, 167–172.
• Lee, K.-F. (1988). On the use of triphone models for continuous speech recognition. JASA, 84(S1), S216–S216.
• Lehiste, I., & Shockey, L. (1972). On the perception of coarticulation effects in English VCV syllables. Journal of Speech, Language, and Hearing Research, 15 (3), 500–506.
• Maekawa, K., Koiso, H., Furui, S., & Isahara, H. (2000). Spontaneous speech corpus of japanese. Proceedings of LREC, 2, 947–952.
• Makhoul, J., & Schwartz, R. (1995). State of the art in continuous speech recognition. Proceedings of the National Academy of Sciences, 92, 9956–9963.
• Manuel, S. (1999). Cross-language studies: relating language-particular coarticulation patterns to other language-particular facts. In W. J. Hardcastle & N. Hewlett (Eds.), Coarticulation: Theory, data and techniques (pp. 179–198). Cambridge, UK: Cambridge University Press.
• Maye, J., Weiss, D. J., & Aslin, R. N. (2008). Statistical phonetic learning in infants: Facilitation and feature generalization. Developmental Science, 11 (1), 122–134.
• Maye, J., Werker, J. F., & Gerken, L. A. (2002). Phonetic details in perception and production allow various patterns in phonological change. Cognition, 82(3), B101–B111.
• McMurray, B., & Aslin, R. (2005). Infants are sensitive to within-category variation in speech perception. Cognition, 95(2), B15–B26.
• McMurray, B., Aslin, R. N., Tanenhaus, M. K., Spivey, M. J., & Subik, D. (2008). Gradient sensitivity to within-category variation in words and syllables. Journal of Experimental Psychology: Human Perception and Performance, 34(6), 1609–1631.
• Mcmurray, B., Aslin, R., & Toscano, J. (2009). Statistical learning of phonetic categories: insights from a computational approach. Developmental Science, 12(3), 369–378.
• Monaghan, P., & Christiansen, M. H. (2010). Words in puddles of sound: modelling psycholinguistic effects in speech segmentation. J. Child Lang., 37(03), 545.
• Nakatani, L. H., & Dukes, K. D. (1977). Locus of segmental cues for word juncture. Journal of the Acoustical Society of America, 62(3), 714–719.
• Ngon, C., Martin, A. T., Dupoux, E., Cabrol, D., Dutat, M., & Peperkamp, S. (In press). Nonwords, nonwords, nonwords: Evidence for a proto-lexicon during the first year of life. Developmental Science.
• Norris, D., McQueen, J., & Cutler, A. (2003). Perceptual learning in speech. Cognitive Psychology, 47(2), 204–238.
• Ohman, S. E. G. (1966). Coarticulation in VCV utterances: Spectrographic measurements. The Journal of the Acoustical Society of America, 39, 151–168.
• Oostdijk, N. (2000). The Spoken Dutch Corpus. Overview and first evaluation. Proceedings of LREC-2000, Athens, 2, 887–894.
• Pegg, J. E., & Werker, J. F. (1997). Adult and infant perception of two English phones. Journal of the Acoustical Society of America, 102(6), 3742–3753.
• Peperkamp, S., & Dupoux, E. (2002). Coping with phonological variation in early lexical acquisition. In I. Lasser (Ed.), The Process of Language Acquisition: Proceedings of the 1999 GALA Conference (pp. 359–385). Frankfurt: Peter Lang.
• Peperkamp, S., Le Calvez, R., Nadal, J., & Dupoux, E. (2006). The acquisition of allophonic rules: Statistical learning with linguistic constraints. Cognition, 101(3), B31–B41.
• Pierrehumbert, J. B. (2003). Phonetic diversity, statistical learning, and acquisition of phonology. Language and Speech, 46(2-3), 115–154.
• Ramus, F., Peperkamp, S., Christophe, A., Jacquemot, C., Kouider, S., & Dupoux, E. (2010). A psycholinguistic perspective on the acquisition of phonology. Papers in Laboratory Phonology, 10, 311–340.
• Regier, T. (2003). Emergent constraints on word-learning: A computational perspective. Trends in Cognitive Sciences, 7(6), 263–268.
• Rytting, C., Brew, C., & Fosler-Lussier, E. (2010). Segmenting words from natural speech: subsegmental variation in segmental cues. Journal of Child Language, 37(3), 513.
• Saffran, J. R., & Thiessen, E. D. (2003). Pattern induction by infant language learners. Developmental Psychology, 39(3), 484–494.
• Seidl, A., Cristia, A., Bernard, A., & Onishi, K. H. (2009). Allophonic and phonemic constrasts in infants’ learning of sound patterns. Language Learning and Development, 5(3), 191–202.
• Sundara, M., Polka, L., & Genesee, F. (2006). Language-experience facilitates discrimination of ⁄ d- ⁄ in monolingual and bilingual acquisition of English. Cognition, 100(2), 369–388.
• Swingley, D. (2005). Statistical clustering and the contents of the infant vocabulary. Cognitive Psychology, 50, 86–132.
• Swingley, D. (2009). Contributions of infant word learning to language development. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1536), 3617–3632.
• Tesar, B., & Smolensky, P. (1998). Learnability in optimality theory. Linguistic Inquiry, 29 (2), 229–268.
• Vallabha, G. K., McClelland, J. L., Pons, F., Werker, J. F., & Amano, S. (2007). Unsupervised learning of vowel categories from infant-directed speech. PNAS, 104(33), 13273–13278.
• Varadarajan, B., Khudanpur, S., & Dupoux, E. (2008). Unsupervised learning of acoustic sub-word units. Proceedings of ACL-08: HLT, Short Papers (Companion Volume), 46, 165–168.
• Venkataraman, A. (2001). A statistical model for word discovery in transcribed speech. Computational Linguistics, 27(3), 351–372.
• Werker, J. F., & Tees, R. (1984). Cross-language speech perception: Evidence for perceptual reorganization during the first year of life. Infant Behavior and Development, 7, 49–63.
• Werker, J. F., & Tees, R. C. (1999). Influences on infant speech processing: Toward a new synthesis. Annual Review of Psychology, 50, 509–535.
• White, K. S., Peperkamp, S., Kirk, C., & Morgan, J. L. (2008). Rapid acquisition of phonological alternations by infants. Cognition, 107(1), 238–265.

Abstract¶

乳児は母国語の音声学的なカテゴリを学習するのと同時期に,流暢な発話から単語のセグメントを学習している. しかし,音声学的なカテゴリの獲得の説明は典型的に音声の中に現れる単語についての情報を無視してきた. 我々は,ベイジアンモデルを使用して,単語のセグメントから,どの程度のフィードバックが音声学的なカテゴリ学習に制約を加え, 学習者が音声学的カテゴリのオーバーラップを明確にするを手助けするのかを例に示す. シミュレーションは人工的なレキシコン由来の情報は英語の母音カテゴリを上手に明確にすることが可能であることを示し, 分布の情報のみの場合と比べ,より頑健なカテゴリ学習を行った.

Infants learn to segment words from fluent speech during the same period as they learn native language phonetic categories, yet accounts of phonetic category acquisition typically ignore information about the words in which speech sounds appear. We use a Bayesian model to illustrate how feedback from segmented words might constrain phonetic category learning, helping a learner disambiguate overlapping phonetic categories. Simulations show that information from an artificial lexicon can successfully disambiguate English vowel categories, leading to more robust category learning than distributional information alone.

Introduction¶

彼らの母国語を学んでいる乳児は,様々なレベルで,知覚空間における音声学的カテゴリの位置や,流暢な発話から彼らは小分けにする単語の同定を含む構造体を抽出する必要がある. 乳児は最初に彼らの言語の音声学的なカテゴリを学習し,ついで,これらのカテゴリを単語のトークンを語彙的なアイテムにマッピングをするヒントに使うという,これらのステップが連続的に生じることは,しばしば暗黙的に当然のこととされている. しかし,乳児は流暢な発話から単語の区分けを,6ヶ月程度から始める(Jusczyk & Aslin, 1995; Jusczyk, Houston, & Newsome, 1999). 非母語発話の弁別は対照的に同じくらいの時期,6-12ヶ月にかけて減衰する(Werker & Tees, 1984). このことは,乳児が言語音と単語の両方をカテゴライズすることを同時に学習している場合における,従来とは異なり,潜在的に2つの学習のプロセスが相互作用する可能性のあるような学習の道筋を示唆している.

Infants learning their native language need to extract several levels of structure, including the locations of phonetic categories in perceptual space and the identities of words they segment from fluent speech. It is often implicitly assumed that these steps occur sequentially, with infants first learning about the phonetic categories in their language and subsequently using those categories to help them map word tokens onto lexical items. However, infants begin to segment words from fluent speech as early as 6 months (Bortfeld, Morgan, Golinkoff, & Rathbun, 2005) and this skill continues to develop over the next several months (Jusczyk & Aslin, 1995; Jusczyk, Houston, & Newsome, 1999). Discrimination of non-native speech sound contrasts declines during the same time period, between 6 and 12 months (Werker & Tees, 1984). This suggests an alternative learning trajectory in which infants simultaneously learn to categorize both speech sounds and words, potentially allowing the two learning processes to interact.

本稿では,我々は乳児が流暢な発話から切り分けるを単語は音声的なカテゴリの獲得のための役立つ情報源を提供することができるという仮説を検討した. 我々は,区切られた単語からの情報をフィードバックしたり音声学的的なカテゴリ学習を抑制したりできる相互作用的なシステムにおける音声学的カテゴリの学習の問題の本質を調査するためにベイジアンアプローチを使用した. 我々の相互作用的モデルは基礎的な語彙と音声学的なすべてのリストを同時に学習し,区切られたトークンの音響的な表現が同じか,違うのか(例えば, bed vs bad)を決定し,語彙的なアイテムは同じ母音を含んでいるのかいないのかを判断する(例えば,send vs act).

In this paper we explore the hypothesis that the words infants segment from fluent speech can provide a useful source of information for phonetic category acquisition. We use a Bayesian approach to explore the nature of the phonetic category learning problem in an interactive system, where information from segmented words can feed back and constrain phonetic category learning. Our interactive model learns a rudimentary lexicon and a phoneme inventory[1] simultaneously, deciding whether acoustic representations of segmented tokens correspond to the same or different lexical items (e.g. bed vs. bad) and whether lexical items contain the same or different vowels (e.g. send vs. act).

シミュレーションでは,セグメントされた単語からの情報を音声学的なカテゴリの獲得を制限するために使用することは,より眼瞼なカテゴリ学習を少ないデータ数からでも可能にし,特に,ある単語が特定の発話音声を含んでいるという情報を使う相互作用的な能力はオーバラップをしたカテゴリの曖昧性をなくすということを実証した.

Simulations demonstrate that using information from segmented words to constrain phonetic category acquisition allows more robust category learning from fewer data points, due to the interactive learner’s ability to use information about which words contain particular speech sounds to disambiguate overlapping categories.

本稿は次のように構成されている. 我々は,我々のモデルのための数学的なフレームワークに関する導入を行う.その後,その定性的な性質を示すためにおもちゃのシミュレーションを示す. 続いてシミュレーションは人工的な語彙からの情報は英語の母音カテゴリを関係する母音のフォルマントをはっきりさせることができることを示す. 最後に,言語獲得のための潜在的な影響について議論し,モデルの解釈を再検討し,後の研究のための方向性を示唆する.

The paper is organized as follows. We begin with an introduction to the mathematical framework for our model, then present toy simulations to demonstrate its qualitative properties. Next, simulations show that information from an artificial lexicon can disambiguate formant values associated with English vowel categories. The last section discusses potential implications for language acquisition, revisits the model’s assumptions, and suggests directions for future research.

Bayesian Model of Phonetic Category Learning¶

音声学的カテゴリの学習を扱っている最近の研究は,頻度学習の重要性に着目してきた. Maye, Werker, and Gerken (2002)は,発話に沿った音声の特別な頻度分布(バイモーダルかユニモーダルか)が, 乳児の連続体の終了点 [1] (多分,特徴量自体は連続して変化するわけだけど,そのカテゴリを知覚するための境界のこと) の弁別に影響することを発見した. つまり,乳児はバイモーダルの分布に親しんだ [2] とき, 境界の弁別がうまく行くことを示したのだ. この業績はガウス混合分布を使用した機械学習にインスパイアされたものであり,音声学的なカテゴリは音声のガウシアン分布,要は正規分布である,として,表現されていることを仮定しており, 学習者は彼らが聞いた音声の分布を最もよく再現できるガウス分布のカテゴリを発見すると仮定している. Boer and Kuhl (2003)はEMアルゴリズム(Dempster, Laird, & Rubin, 1977)をフォルマントデータから木構造の位置の学習を行うために使用している. McMurray, Aslin, and Toscano (2009) EMアルゴリズムに似た最急降下法 [3] を有声子音の閉鎖子音の学習に導入し,このアルゴリズムは母音と子音両方のデータ用に多次元に拡張された(Toscano & McMurray, 2008; Vallabha, McClelland, Pons, Werker, & Amano, 2007).

Recent research on phonetic category acquisition has focused on the importance of distributional learning. Maye, Werker, and Gerken (2002) found that the specific frequency distribution (bimodal or unimodal) of speech sounds along a continuum could affect infants’ discrimination of the continuum endpoints, with infants showing better discrimination of the endpoints when familiarized with the bimodal distribution. This work has inspired computational models that use a Mixture of Gaussians approach, assuming that phonetic categories are represented as Gaussian, or normal, distributions of speech sounds and that learners find the set of Gaussian categories that best represents the distribution of speech sounds they hear. Boer and Kuhl (2003) used the Expectation Maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977) to learn the locations of three such vowel categories from formant data. McMurray, Aslin, and Toscano (2009) introduced a gradient descent algorithm similar to EM to learn a stop consonant voicing contrast, and this algorithm has been extended to multiple dimensions for both consonant and vowel data (Toscano & McMurray, 2008; Vallabha, McClelland, Pons, Werker, & Amano, 2007).

我々のモデルはガウス混合アプローチを上記の先行モデルを採用したが,ノンパラメトリックなベイジアンのフレームワークを使用した. このフレームワークはモデルを単語レベルまで拡張し,構造体の複数のレベルを操作する際に,学習結果を調査することを可能にする. 先行モデルのように,我々のモデルにおける発話音声は安定状態のフォルマントやVOTなどの音声学的次元を使って再現された. 単語は音声学的な値 [4] の連続体で,そこは,それぞれの音素の音声学的な値の一つの非連続体のセット(例えば第一,第二フォルマント)と一致する場所である. Toy corpusの一つのフラグメント [5] を図1に示す. 発話音声を使用した音声学的な一覧 [6] は4つのカテゴリを持っており,A,B,C,Dというラベルが振られている. つまり,5つの単語が示されており,それぞれ,ADA,AB,D,AB,DCという語彙的な要素を表現している. 学習は,発話音声を使用したものと,相互作用的学習者の場合は,単語において,他の音のどれが現れるのかについての情報を含んでおり,コーパスを発生させた音声学的カテゴリを復元することが目的である.

Our model adopts the Mixture of Gaussians approach from these previous models but uses a non-parametric Bayesian framework that allows extension of the model to the word level, making it possible to investigate the learning outcome when multiple levels of structure interact. As in previous models, speech sounds in our model are represented using phonetic dimensions such as steady-state formant values or voice onset time. Words are sequences of these phonetic values, where each phoneme corresponds to a single discrete set (e.g. first and second formant) of phonetic values. A sample fragment of a toy corpus is shown in Figure 1. The phoneme inventory has four categories, labeled A, B, C, and D; five words are shown, representing lexical items ADA, AB, D, AB, and DC, respectively. Learning involves using the speech sounds and, in the case of an interactive learner, information about which other sounds appear with them in words, to recover the phonetic categories that generated the corpus.

シミュレーションでは2つのモデルを比較した. これらは,学習者に割り当てる仮説空間が異なる. 分布モデルでは,学習者の仮説空間は音素一覧に含まれており,そこは,音声学的空間における発話音声のガウス分布が音素と一致する. 語彙-分布モデルでは,学習者は上記と同じ音素一覧を考えるが,それらの音素一覧は音素の連続体からなる語彙的な要素を含む目録とのみ結合すると考える. そのため,語彙-分布モデル学習者は,音声学的カテゴリのセットの復元に音声学的情報のみでなく,それらの音を含む単語についての情報も使用できる.

Simulations compare two models that differ in the hypothesis space they assign to the learner. In the distributional model, the learner’s hypothesis space contains phoneme inventories, where phonemes correspond to Gaussian distributions of speech sounds in phonetic space. In the lexical-distributional model, the learner considers these same phoneme inventories, but considers them only in conjunction with lexicons that contain lexical items composed of sequences of phonemes. This allows the lexical-distributional learner to use not only phonetic information, but also information about the words that contain those sounds, in recovering a set of phonetic categories.

訳者注

Distributional Model¶

分布モデルにおいては,学習者は我々が,音素イベントCと呼ぶ音韻のセットをコーパスの音声から復元する必要がある [7] . このモデルは単語,単語境界に関するすべての情報を無視し,音韻空間の発話音声の分布からのみ学習を行う. 発話音声は音韻イベントリから音素カテゴリ \(C\) を選択することによって生成されると仮定し, そのカテゴリに関連するガウシアン分布から,音韻の値 [#f42]_ を抽出している. カテゴリは,それらのデータの平均値 \(\mu_c\) ではなく, 共分散行列 \(\Sigma_c\) であり,発生頻度である. 以下の先行研究,形態論の先行研究(Goldwater, Griffiths, & Johnson, 2006),単語のセグメント(Goldwater, Griffiths, & Johnson, in press),そして文法の学習(Johnson, Griffiths, & Goldwater, 2007)では, 学習者の音素のイベントに関する事前知識はDirichlet process(Ferguson, 1973)と呼ばれるノンパラメトリックなベイズモデルを使用して実装されている. この分布は音素一覧のカテゴリ数に対するバイアス [#f43]_ と,これらのカテゴリの音響的パラメータに対するバイアスをエンコードしたものである. 音韻的カテゴリの数についての事前知識は, 学習者が潜在的に莫大な数のカテゴリ数を考慮することを可能にするが, 少ない数のカテゴリへと向かわせるバイアスを提供する.バイアスの強さについては パラメータ \(\alpha\) により制御されている. これは先行モデル(McMurray et al., 2009; Vallabha et al., 2007)で使用された, カテゴリの割り当てにおける”勝者総取りバイアス”に置き換えることができ, データを表現するのに必要なカテゴリの数の明示的な推定を可能にする.

In the distributional model, a learner is responsible for recovering a set of phonetic categories, which we refer to as a phoneme inventory \(C\), from a corpus of speech sounds. The model ignores all information about words and word boundaries, and learns only from the distribution of speech sounds in phonetic space. Speech sounds are assumed to be produced by selecting a phonetic category \(c\) from the phoneme inventory and then sampling a phonetic value from the Gaussian associated with that category. Categories differ in their means \(\mu_c\) , covariance matrices \(\Sigma_c\) , and frequencies of occurrence. Following previous work in morphology (Goldwater, Griffiths, & Johnson, 2006), word segmentation (Goldwater, Griffiths, & Johnson, in press), and grammar learning (Johnson, Griffiths, & Goldwater, 2007), learners’ prior beliefs about the phoneme inventory are encoded using a nonparametric Bayesian model called the Dirichlet process (Ferguson, 1973), \(C∼DP(\alpha, G_C)\). This distribution encodes biases over the number of categories in the phoneme inventory, as well as over phonetic parameters for those categories. Prior beliefs about the number of phonetic categories allow the learner to consider a potentially infinite number of categories, but produce a bias toward fewer categories, with the strength of the bias controlled by the parameter \(\alpha\).[2] This replaces the winner-take-all bias in category assignments that has been used in previous models (McMurray et al., 2009; Vallabha et al., 2007) and allows explicit inference of the number of categories needed to represent the data.

訳者注

音韻パラメータの事前分布は \(G_C\) によって定義され,このモデルにおいては,カテゴリの分散 \(\Sigma_c∼IW(\nu_0,\Sigma_0 )\) に対する逆ウィシャート事前分布とカテゴリの平均値 \(\mu_c \mid \Sigma_c ∼ N(\mu_0 , {\Sigma_c \over \nu_0} )\) に対するガウシアン事前分布を含んでいる,ガウシアンな音韻カテゴリに対するものである. これらの分布のパラメータは擬似データの \(\mu_0\) , \(\Sigma_0\) , \(\nu_0\) が平均,共分散及び学習者がすでに新しいカテゴリに割り当てられたと想像する音声の数をどこでエンコードするのかを考えることが可能です. この音韻パラメータに対する事前分布は論理モデルの中心ではなく,計算を簡単にするための処理である.擬似データにおける発話音声の数は可能な限り少なくしている[3]ため,事前バイアスはリアルデータによって書き換えられることになる. 音響的な値の連続値を提示することで,学習者はこれらの音響的な値から発生するガウシアンのカテゴリセットを復元することが必要である. マルコフ連鎖モンテカルロ法の形で,ギブズサンプリング (Geman & Geman, 1984) は理想的な学習者がコーパスを生成した可能性が高いと考えている音素の目録の例を復元するために使用された. 発話音声ははじめはランダムな割り当てで与えら得ており,各sweepの間で,コーパスを通じて,順番にすべての現在の割り当てに基づいた新しいカテゴリの割り当てが与えられる.

The prior distribution over phonetic parameters is defined by \(G_C\) , which in this model is a distribution over Gaussian phonetic categories that includes an Inverse-Wishart prior over category variances, \(\Sigma_c∼IW(\nu_0,\Sigma_0 )\), and a Gaussian prior over category means, \(\mu_c \mid \Sigma_c ∼ N(\mu_0 , {\Sigma_c \over \nu_0} )\). The parameters of these distributions can be thought of as pseudo data, where \(\mu_0\) , \(\Sigma_0\) , and \(\nu_0\) encode the mean, covariance, and number of speech sounds that the learner imagines having already assigned to any new category. This prior distribution over phonetic parameters is not central to the theoretical model, but rather is included for ease of computation; the number of speech sounds in the pseudodata is made as small as possible[3] so that the prior biases are overshadowed by real data. Presented with a sequence of acoustic values, the learner needs to recover the set of Gaussian categories that generated those acoustic values. Gibbs sampling (Geman & Geman, 1984), a form of Markov chain Monte Carlo, is used to recover examples of phoneme inventories that an ideal learner believes are likely to have generated the corpus. Speech sounds are initially given random category assignments, and in each sweep through the corpus, each speech sound in turn is given a new category assignment based on all the other current assignments. The probability of assignment to category \(c\) is given by Bayes’ rule,

where wi j denotes the phonetic parameters of the speech sound in position j of word i. The prior p(c) is given by the Dirichlet process and is

making it proportional to the number of speech sounds \(n_c\) already assigned to that category, with some probability \(\alpha\) of assignment to a new category. The likelihood \(p(w_{ij} \mid c)\) is obtained by integrating over all possible means and covariance matrices for category \(c\) , \(\int\int p(w_{ij} \mid \mu_c , \sum_c)p(\mu_c \mid\sum_c )p(\sum_c )d\mu_c d\sum_c\) , where the probability distributions \(p(\mu_c \mid\sum_c)\) and \(p(\sum_c)\) are modified to take into account the speech sounds already assigned to that category.

This likelihood function has the form of a multivariate tdistribution and is discussed in more detail in Gelman, Carlin, Stern, and Rubin (1995). Using this procedure, category assignments converge to the posterior distribution on phoneme inventories, revealing an ideal learner’s beliefs about which categories generated the corpus.

Lexical-Distributional Model¶

This non-parametric Bayesian framework has the advantage that it is straightforward to extend to hierarchical structures (Teh, Jordan, Beal, & Blei, 2006), allowing us to explore the influence of words on phonetic category acquisition. In the lexical-distributional model, the learner recovers not only the same phoneme inventory C as in the distributional model, but also a lexicon L with lexical items composed of sequences of phonemes. This creates an extra step in the generative process: instead of assuming that the phoneme inventory generates a corpus directly, as in the distributional model, this model assumes that the phoneme inventory generates the lexicon and that the lexicon generates the corpus. The corpus is generated by selecting a lexical item to produce and then sampling an acoustic value from each of the phonetic categories contained in that lexical item.

The prior probability distribution over possible lexicons is a second Dirichlet process, L ∼ DP(β, GL ) where GL defines a prior distribution over lexical items. This prior favors shorter lexical items, assuming word lengths to be generated from a geometric distribution, and assumes that a category for each phoneme slot has been sampled from the phoneme inventory C. Thus, the prior probability distribution over words is defined according to the phoneme inventory, and the learner needs to optimize the phoneme inventory so that it generates the lexicon. Parallel to the bias toward fewer phonetic categories, the model encodes a bias toward fewer lexical items but allows a potentially infinite number of lexical items.

Presented with a corpus consisting of isolated word tokens, each of which consists of a sequence of acoustic values, the language learner needs to recover the lexicon and phoneme inventory of the language that generated the corpus. Learning is again performed through Gibbs sampling. Each iteration now includes two sweeps: one through the corpus, assigning each word to the lexical item that generated it, and one through the lexicon, assigning each position of each lexical item to its corresponding phoneme from the phoneme inventory. In the first sweep we use Bayes’ rule to calculate the probability that word wi corresponds to lexical item k,

p(k|wi ) proptop(wi midk)p(k) (3)

Parallel to Equation 2, the prior is

nk / ∑ k_n_k +β for existing categories

p(k) = (4)

β / ∑ k_n_k +β for a new category

where nk is the number of word tokens already assigned to lexical item k. A word is therefore assigned to a lexical item with a probability proportional to the number of times that lexical item has already been seen, with some probability β reserved for the possibility of seeing a new lexical item. The likelihood is a product of the likelihoods of each speech sound having been generated from its respective category,

p(wi midk) = ∏ p(wi j midck j ) (5)

j

where j indexes a particular position in the word and ck j is the phonetic category that corresponds to position j of lexical item k. Any lexical item with a different length from the word wi is given a likelihood of zero, and samples from the prior distribution on lexical items are used to estimate the likelihood of a new lexical item (Neal, 1998). The second sweep uses Bayes’ rule

p(cmidw{k} j ) proptop(w{k} j midc)p(c) (6)

to assign a phonetic category to position j of lexical item k, where w{k} j is the set of phonetic values at position j in all of the words in the corpus that have been assigned to lexical item k. The prior p(c) is the same prior over category assignments as was used in the distributional model, and is given by Equation 2. The likelihood p(w{k} j midc) is again computed by integrating over all possible means and covariance maRR trices, FF ∏ wi ∈ k p(wi j midmuc , Sigma c )p(muc midSigma c )p(Sigma c )dmuc dSigma c , this time taking into account phonetic values from all the words assigned to lexical item k. The sampling procedure converges on samples from the joint posterior distribution on lexicons and phoneme inventories, allowing learners to recover both levels of structure simultaneously.

Qualitative Behavior of an Interactive Learner¶

このセクションでは,Toy simulation [10] がどの語彙が提供するのか

In this section, toy simulations demonstrate how a lexicon can provide disambiguating information about overlapping categories that would be interpreted as a single category by a purely distributional learner. We show that it is not the simple presence of a lexicon, but rather specific disambiguating information within the lexicon, that increases the robustness of category learning in the lexical-distributional learner. Corpora were constructed for these simulations using four categories labeled A, B, C, and D, whose means are located at -5, -1, 1, and 5 along an arbitrary phonetic dimension (Figure 2 (a)). All four categories have a variance of 1. Because the means of categories B and C are so close together, being separated by only two standard deviations, the overall distribution of tokens in these two categories is unimodal. To test the distributional learner, 1200 acoustic values were sampled from these categories, with 400 acoustic values sampled from each of Categories A and D and 200 acoustic values sampled from each of Categories B and C. Results indicate that these distributional data are not strong enough to disambiguate categories B and C, leading the learner to interpret them as a single category (Figure 2 (b)).[4] While this may be due in part to the distributional learner’s prior bias toward fewer categories, simulations in the next section will show that the gradient descent learner from Vallabha et al. (2007), which has no such explicit bias, shows similar behavior.

訳者注

Two toy corpora were constructed for the lexicaldistributional model from the 1200 phonetic values sampled above. The corpora differed from each other only in the distribution of these values across lexical items. The lexicon of the first corpus contained no disambiguating information about speech sounds B and C. It was generated from six lexical items, with identities AB, AC, DB, DC, ADA, and D. Each lexical item was repeated 100 times in the corpus for a total of 600 word tokens. In this corpus, Categories B and C appeared only in minimal pair contexts, since both AB and AC, as well as both DB and DC, were words. As shown in Figure 2 (c), the lexical-distributional learner merged categories B and C when trained on this corpus. Merging the two categories allowed the learner to condense AB and AC into a single lexical item, and the same happened for DB and DC. Because the distribution of these speech sounds in lexical items was identical, lexical information could not help disambiguate the categories.

The second corpus contained disambiguating information about categories B and C. This corpus was identical to the first except that the acoustic values representing the phonemes B and C of words AC and DB were swapped, converting these words into AB and DC, respectively. Thus, the second corpus contained only four lexical items, AB, DC, ADA, and D, and there were now 200 tokens of words AB and DC. Categories B and C did not appear in minimal pair contexts, as there was a word AB but no word AC, and there was a word DC but no word DB. The lexical-distributional learner was able to use the information contained in the lexicon in the second corpus to successfully disambiguate categories B and C (Figure 2 (d)). This occurred because the learner could categorize words AB and DC as two different lexical items simply by recognizing the difference between categories A and D, and could use those lexical classifications to notice small phonetic differences between the second phonemes in these lexical items.

In this model it is non-minimal pairs, rather than minimal pairs, that help the lexical-distributional learner disambiguate phonetic categories. While minimal pairs may be useful when a learner knows that two similar sounding tokens have different referents, they pose a problem in this model because the learner hypothesizes that similar sounding tokens represent the same word. Thiessen (2007) has made a similar observation with 15-month-olds in a word learning task, showing that infants may fail to notice a difference between similarsounding object labels, but are better at discriminating these words when familiarized with non-minimal pairs that contain the same sounds.

Learning English Vowels¶

自然言語におけるカテゴリのオーバーラップの典型例は母音のカテゴリである. 例えば,Hillenbrand, Getty, Clark, and Wheeler (1995)らは英語の母音に関して図4(a)を示している. したがって我々は英語の母音カテゴリを語彙的な分布学習者の実際の音韻カテゴリパラメータを基本にしたオーヴァラップをカテゴリの曖昧さをなくす能力をテストするために英語の母音を使用した.

The prototypical examples of overlapping categories in natural language are vowel categories, such as the English vowel categories from Hillenbrand, Getty, Clark, and Wheeler (1995) shown in Figure 4 (a).[5] We therefore use English vowel categories to test the lexical-distributional learner’s ability to disambiguate overlapping categories that are based on actual phonetic category parameters.

Hillenbrand et al. (1995)の母音フォルマントデータを基本にした音韻カテゴリを使用して2つのコーパスを作成した. 最初のコーパスのカテゴリは男性によって発話された母音をベースにしており,適度なオーバーラップしかない(図3 (a)). 2つ目のコーパスのカテゴリは男性,女性,子供によって発話された母音をベースにしており,オーバーラップの程度が大きい(図4 (a)). どちらのケースでも,12音素のカテゴリの平均と共分散行列は対応する母音トークンから算出した. それぞれのコーパスのために発生モデルを使用して,母音のみからなる語彙項目の仮想的な集合を作成し,トークンをガウスカテゴリパラメータの適当なセットから,この語彙をベースに5000語のトークンを作成した.

Two corpora were constructed using phonetic categories based on the Hillenbrand et al. (1995) vowel formant data. Categories in the first corpus were based on vowels spoken by men, and had only moderate overlap (Figure 3 (a)); categories in the second corpus were based on vowels spoken by men, women, and children, and had a much higher degree of overlap (Figure 4 (a)). In each case, means and covariance matrices for the twelve phonetic categories were computed from corresponding vowel tokens. Using the generative model, a hypothetical set of lexical items consisting only of vowels was generated for each corpus, and 5,000 word tokens were generated based on this lexicon from the appropriate set of Gaussian category parameters.

これらのコーパスは以下の3つのモデルにテストデータとして与えられた.

• 語彙-分布モデル
• 分布モデル
• Vallabha et al.(2007)で使用された多次元勾配降下アルゴリズム[6]

男性発話をベースにしたコーパスを使用した結果を図3に示す. また,すべての話者の発話をベースにしたコーパスへの結果は図4である. それぞれの場合で,語彙-分布モデルは母音カテゴリのセットの復元に成功し,曖昧な近隣との境界推定に成功した. 一方,語彙がかけたモデルでは近隣の母音カテゴリのペアをいくつか誤ってマージしてしまった. そのため,語彙を仮定することは,語彙に含まれる音韻フォームが明示的に学習者に与えなくとも,学習者の母音カテゴリの重複の曖昧さをなくすことを助けることを示す証拠になる.

These corpora were given as training data to three models: the lexical-distributional model, the distributional model, and the multidimensional gradient descent algorithm used by Vallabha et al. (2007).[6] Results for the corpus based on men’s productions are shown in Figure 3, and results from the corpus based on all speakers’ productions are shown in Figure 4. In each case, the lexical-distributional learner recovered the correct set of vowel categories and successfully disambiguated neighboring categories. In contrast, the models lacking a lexicon mistakenly merged several pairs of neigh boring vowel categories. Positing the presence of a lexicon therefore showed evidence of helping the ideal learner disambiguate overlapping vowel categories, even though the phonological forms contained in the lexicon were not given explicitly to the learner.

ペアごとの正確性と完全性の対策は、モデルの性能（表1）の定量的尺度として、各学習者に対して計算された. これらの尺度では,正しく同じカテゴリーに入れた母音のトークンのペアを、ヒットとしてカウントし,同じカテゴリにされているべきときに、誤って別のカテゴリに割り当てられたトークンのペアはミスとしてカウントした. また異なるカテゴリにされているべきときに、誤って同じカテゴリに割り当てられたトークンのペアは、誤警報としてカウントした.

Pairwise accuracy and completeness measures were computed for each learner as a quantitative measure of model performance (Table 1). For these measures, pairs of vowel tokens that were correctly placed into the same category were counted as a hit; pairs of tokens that were incorrectly assigned to different categories when they should have been in the same category were counted as a miss; and pairs of tokens that were incorrectly assigned to the same category when they should have been in different categories were counted as a false alarm.

accuracyの得点は”hits/ hits + false alarms”で計算し,completenessは”hits/hits+misses”として計算した. 両方の基準で語彙-分布モデルは高い点数を出したが,誤って複数の重複するカテゴリを合併しているという事実を反映して、accuracyの得点は実質的に純粋に分布だけから学習した場合よりも低かった.

The accuracy score was computed a hits/ hits + false alarms and the completeness score as hits/hits+misses . Both measures were high for the lexical-distributional learner, but accuracy scores were substantially lower for the purely distributional learners, reflecting the fact that these models mistakenly merged several overlapping categories.

この結果は,予測されるように,音韻のカテゴリに加えて、単語のカテゴリを学習に使用するモデルは、音素のカテゴリを学習したモデルよりも優れた表音カテゴリの学習結果が得られることを示している. 分布のモデルの学習者は、ちょうど最初の2フォルマントを超え次元を与えたりしている場合(Vallabha et al., 2007)や、学習中に、より多くのデータ·ポイントを与えられている場合は,より良い性能を示す可能性があることに注目してほしい. これら2つのソリューションは確かに,お互いに対して機能する.つまり,デモンストレーションを加えるごとに,おなじ学習結果を維持するのに必要なデータ数が増えていく. しかし,我々は純粋な分布の学習モデルは音韻のカテゴリを取得できないことを示唆する気はない. ここで紹介したシミュレーションは音韻カテゴリに実質的にオーバーラップのある言語において,学習者が特定の言語音を含む単語情報を使用できるインタラクティブなシステムは,表音カテゴリ学習の頑健さを高めることができることを実証するものである.

Results suggest that as predicted, a model that uses the input to learn word categories in addition to phonetic categories produces better phonetic category learning results than a model that only learns phonetic categories. Note that the distributional learners are likely to show better performance if they are given dimensions beyond just the first two formants (Vallabha et al., 2007) or if they are given more data points during learning. These two solutions actually work against each other: as dimensions are added, more data are necessary to maintain the same learning outcome. Nevertheless, we do not wish to suggest that a purely distributional learner cannot acquire phonetic categories. The simulations presented here are instead meant to demonstrate that in a language where phonetic categories have substantial overlap, an interactive system, where learners can use information from words that contain particular speech sounds, can increase the robustness of phonetic category learning.

Discussion¶

This paper has presented a model of phonetic category acquisition that allows interaction between speech sound and word categorization. The model was not given a lexicon a priori, but was allowed to begin learning a lexicon from the data at the same time that it was learning to categorize individual speech sounds, allowing it to take into account the distribution of speech sounds in words. This lexical-distributional learner outperformed a purely distributional learner on a corpus whose categories were based on English vowel categories, showing better disambiguation of overlapping categories from the same number of data points.

Infants learn to segment words from fluent speech around the same time that they begin to show signs of acquiring native language phonetic categories, and they are able to map these segmented words onto tokens heard in isolation (Jusczyk & Aslin, 1995), suggesting that they are performing some sort of rudimentary categorization on the words they hear. Infants may therefore have access to information from words that can help them disambiguate overlapping categories. If information from words can feed back to constrain phonetic category learning, the large degree of overlap be tween phonetic categories may not be such a challenge as is often supposed.

In generalizing these results to more realistic learning situations, however, it is important to take note of two simplifying assumptions that were present in our model. The first key assumption is that speech sounds in phonetic categories follow the same Gaussian distribution regardless of phonetic or lexical context. In actual speech data, acoustic characteristics of sounds change in a context-dependent manner due to coarticulation with neighboring sounds (e.g. Hillenbrand, Clark, & Nearey, 2001). A lexical-distributional learner hearing reliable differences between sounds in different words might erroneously assign coarticulatory variants of the same phoneme to different categories, having no other mechanism to deal with context-dependent variability. Such variability may need to be represented explicitly if an interactive learner is to categorize coarticulatory variants together.

A second assumption concerns the lexicon used in the vowel simulations, which was generated from our model. Generating a lexicon from the model ensured that the learner’s expectations about the lexicon matched the structure of the lexicon being learned, and allowed us to examine the influence of lexical information in the best case scenario. However, several aspects of the lexicon, such as the assumption that phonemes in lexical items are selected independently of their neighbors, are unrealistic for natural language. In future work we hope to extend the present results using a lexicon based on child-directed speech.

Infants learn multiple levels of linguistic structure, and it is often implicitly assumed that these levels of structure are acquired sequentially. This paper has instead investigated the optimal learning outcome in an interactive system using a non-parametric Bayesian framework that permits simultaneous learning at multiple levels. Our results demonstrate that information from words can lead to more robust learning of phonetic categories, providing one example of how such interaction between domains might help make the learning problem more tractable.

Acknowledgments.¶

This research was supported by NSF grant BCS-0631518, AFOSR grant FA9550-07-1-0351, and NIH grant HD32005. We thank Joseph Williams for help in working out the model and Sheila Blumstein, Adam Darlow, Sharon Goldwater, Mark Johnson, and members of the computational modeling reading group for helpful comments and discussion.

References¶

Boer, B. de, & Kuhl, P. K. (2003). Investigating the role of infantdirected speech with a computer model. Acoustics Research Letters Online, 4(4), 129-134.

Bortfeld, H., Morgan, J. L., Golinkoff, R. M., & Rathbun, K. (2005).

Mommy and me: Familiar names help launch babies into speechstream segmentation. Psychological Science, 16(4), 298-304.

Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, B, 39, 1-38.

Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1(2), 209-230. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (1995). Bayesian data analysis. New York: Chapman and Hall.

Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE-PAMI, 6, 721-741.

Goldwater, S., Griffiths, T. L., & Johnson, M. (2006). Interpolating between types and tokens by estimating power-law generators. Advances in Neural Information Processing Systems 18.

Goldwater, S., Griffiths, T. L., & Johnson, M. (in press). A Bayesian framework for word segmentation: Exploring the effects of context. Cognition.

Hillenbrand, J., Getty, L. A., Clark, M. J., & Wheeler, K. (1995). Acoustic characteristics of American English vowels. Journal of the Acoustical Society of America, 97(5), 3099-3111.

Hillenbrand, J. L., Clark, M. J., & Nearey, T. M. (2001). Effects of consonant environment on vowel formant patterns. Journal of the Acoustical Society of America, 109(2), 748-763.

Johnson, M., Griffiths, T. L., & Goldwater, S. (2007). Adaptor grammars: a framework for specifying compositional nonparametric Bayesian models. Advances in Neural Information Processing Systems 19.

Jusczyk, P. W., & Aslin, R. N. (1995). Infants’ detection of the sound patterns of words in fluent speech. Cognitive Psychology, 29, 1-23.

Jusczyk, P. W., Houston, D. M., & Newsome, M. (1999). The beginnings of word segmentation in English-learning infants. Cognitive Psychology, 39, 159-207.

Maye, J., Werker, J. F., & Gerken, L. (2002). Infant sensitivity to distributional information can affect phonetic discrimination. Cognition, 82, B101-B111.

McMurray, B., Aslin, R. N., & Toscano, J. C. (2009). Statistical learning of phonetic categories: Computational insights and limitations. Developmental Science, 12(3), 369-378.

Neal, R. M. (1998). Markov chain sampling methods for Dirichlet process mixture models. Technical Report No. 9815, Department of Statistics, University of Toronto.

Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006). Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101, 1566-1581.

Thiessen, E. D. (2007). The effect of distributional information on children’s use of phonemic contrasts. Journal of Memory and Language, 56(1), 16-34.

Toscano, J. C., & McMurray, B. (2008). Using the distributional statistics of speech sounds for weighting and integrating acoustic cues. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (p. 433-438). Austin, TX: Cognitive Science Society.

Vallabha, G. K., McClelland, J. L., Pons, F., Werker, J. F., & Amano, S. (2007). Unsupervised learning of vowel categories from infantdirected speech. Proceedings of the National Academy of Sciences, 104, 13273-13278.

Werker, J. F., & Tees, R. C. (1984). Cross-language speech perception: Evidence for perceptual reorganization during the first year of life. Infant Behavior and Development, 7, 49-63.

Abstract (要約)¶

Various aspects of motherese also known as infant-directed speech (IDS) have been studied for many years. As it is a widespread phenomenon, it is suspected to play some important roles in infant development. Therefore, our purpose was to provide an update of the evidence accumulated by reviewing all of the empirical or experimental studies that have been published since 1966 on IDS driving factors and impacts. Two databases were screened and 144 relevant studies were retained. General linguistic and prosodic characteristics of IDS were found in a variety of languages, and IDS was not restricted to mothers. IDS varied with factors associated with the caregiver (e.g., cultural, psychological and physiological) and the infant (e.g., reactivity and interactive feedback). IDS promoted infants’ affect, attention and language learning. Cognitive aspects of IDS have been widely studied whereas affective ones still need to be developed. However, during interactions, the following two observations were notable:

• 1. IDS prosody reflects emotional charges and meets infants’ preferences,
• and (2) mother-infant contingency and synchrony are crucial for IDS production and prolongation.

Thus, IDS is part of an interactive loop that may play an important role in infants’ cognitive and social development.

Introduction (導入)¶

Motherese, also known as infant-directed speech (IDS) or “baby-talk”, refers to the spontaneous way in which mothers, fathers, and caregivers speak with infants and young children. In a review of the various terms used to denote young children’s language environments, Saxton suggested the preferential use of “infant or child-directed speech” [1] . In 1964, a linguist [2] defined “baby-talk” as “a linguistic subsystem regarded by a speech community as being primarily appropriate for talking to young children”. He reported that “baby talk” was a well-known, special form of speech that occurred in a number of languages and included the following 3 characteristics:

• 1. intonational and paralinguistic phenomena (e.g., a higher overall pitch),
• 2. words and constructions derived from the normal language (e.g., the use of third person constructions to replace first and second person constructions),
• 3. a set of lexical items that are specific for baby talk.

He provided a precise, documented study of IDS across several different languages. Since then, infant-directed speech has been studied extensively across a number of interactive situations and contexts, especially by researchers interested in understanding language acquisition. A recent review of “babytalk” literature focused on phonological, lexical and syntactic aspects of the input provided to infants from the perspective of language acquisition and comprehension [3] . Although Snow, in a review of the early literature on motherese [4] , claimed that “language acquisition is the result of a process of interaction between mother and child, which begins early in infancy, to which the child makes as important a contribution as the mother, and which is crucial to cognitive and emotional development as well as language acquisition”, few experimental findings have sustained this assertion. Recent progresses in cognitive science and in interactional perspective suggest, however, that infant cognitive development is linked with social interaction (e.g., Kuhl et al., 2003). Motherese could be a crossroad for such a linkage. Here, we aim to review the available evidence relevant to motherese from an interactional perspective, with a specific focus on children younger than 2 years of age. In contrast with Soderstrom’s review (2007), we focus more preferentially on motherese’s prosodic and affective aspects to determine the factors, including interactive ones, associated with its production and variations, its known effects on infants and its suspected functions aside from language acquisition.

Methods (方法)¶

We searched the PubMed and PsycInfo databases from January 1966 to March 2011 using the following criteria: journal article or book chapter with ‘‘motherese’’ or ‘‘infant-directed speech’’ within the title or abstract, published in the English language and limited to human subjects. A diagram summarizing the literature search process is provided in :num:`Figure #diagram-flow` . We found 90 papers with PubMed and 134 with PsycInfo, of which 59 were shared across the databases, for a total of 165 papers. We excluded 50 papers because 11 were reviews or essays and 39 were experimental studies that did not aim to improve knowledge on IDS as they addressed other aims (see details in Annex S1). We found an additional 29 references by screening the reference lists of the 115 papers, leading to a total of 144 relevant papers.

Diagram flow of the literature search.

• doi: 10.1371/journal.pone.0078103.g001

Results (結果)¶

Discussion (ディスカッション)¶

Conclusion (結論)¶

Some authors held the perspective that, beyond language acquisition, IDS significantly influences infants’ cognitive and emotional development (e.g., [4] [176] ). Our systematic review supports this view. More studies are needed to understand how IDS impacts affective factors in infants and how this is linked with infants’ cognitive development, however. An interesting approach may be to investigate how this process is altered by infants’ communicative difficulties, such as early signs of autism spectrum disorder, and how these alterations may affect infants’ development [177] .

Summary of the motherese interactive loop (a) and its socio-cognitive implications (2B).

• 1A: The motherese interactive loop implies that motherese is both a vector and a reflection of mother-infant interaction.
• 2B: Motherese affects intersubjective construction and learning. Its implications for infants’ early socio-cognitive development are evident in affect transmission and sharing, and in infants’ preferences, engagement, attention, learning and language acquisition.

doi: 10.1371/journal.pone.0078103.g002

References (参照)¶

affiliation

abstract¶

To date, the intonation of infant-directed speech (IDS) has been analyzed without reference to its phonological structure. Intonational phonology should, however, inform IDS research, discovering important properties that have previously been overlooked. The present study investigated “intonational exaggeration” in Japanese IDS using the intonational phonological framework. Although intonational exaggeration, which is most often measured by pitch-range expansion, is one of the best-known characteristics of IDS, Japanese has been reported to lack such exaggeration. The present results demonstrated that intonational exaggeration is in fact present and observed most notably at the location of boundary pitch movements, and that the effects of lexical pitch accents in the remainder of the utterances superficially mask the exaggeration. These results not only reveal dynamic aspects of Japanese IDS, but also in turn contribute to the theory of intonational phonology, suggesting that paralinguistic pitch-range modifications most clearly emerge where the intonation system of a language allows maximum flexibility in varying intonational contours.

I.INTRODUCTION¶

II.THE JAPANESE INTONATION SYSTEM¶

A.The phonology of Japanese intonation¶

The description of the Japanese intonation system in this paper is based on the framework called X-JToBI (Maekawa et al., 2002). It is an extended version of the original Japanese Tone and Break Indices, or J_ToBI, framework (Venditti, 2005), which owes its theoretical foundation to the major study of Japanese intonation by Pierrehumbert and Beckman (1988). In this section, we will describe two aspects of Japanese intonation that are relevant to the present study: (1) Lexical pitch accent (and related phenomena) and (2) boundary pitch movement (BPM).

1.Lexical pitch accent and related phenomena¶

Japanese has two types of words in its lexicon generally referred to as “accented words” and “unaccented words”; the former exhibit pitch contours with a steep F0 fall, or “pitch accent,” while the latter show contours with no such fall. The presence or absence of pitch accent and its location in the word are lexically contrastive (Pierrehumbert and Beckman, 1988; Kubozono, 1993). Figure 1 provides an example for accented and unaccented words. In the text, the accented vowel is marked with an apostrophe after it.

Prosodic phrasing occurs at two levels in Japanese. The lower level is the accentual phrase (AP) and the higher one is the intonation phrase (IP). An AP is defined (1) as having a delimitative rise to high around the second mora and a subsequent gradual fall to low at the end of the phrase and (2) as having at most one lexical pitch accent. The tonal pattern of an unaccented AP (i.e., an AP having no accented word) is transcribed as %L H- L% in J_ToBI, where H- is a phrasal high tone around the second mora and L% is a low boundary tone at the end of the phrase. The first %L tone is a low boundary tone that appears only when the AP follows a pause (Venditti, 2005). In Fig. 1 (left), the adjective amai “sweet” is combined with the following noun ame “candy” into a single unaccented AP, where the %L H- L% pattern can be clearly observed.

When an AP contains a lexically accented word, the tonal pattern marking the AP shows a fall on the accented mora. The pattern of such an accented AP is then transcribed as %L (H-) H*þL L%, where H*þL stands for the lexical pitch accent. (In the case of an accented AP, H- is labeled only when the phrasal high tone is distinguishable from the high of the lexical pitch accent.) This pattern can be observed in Fig. 1 (right), where the lexically accented verb uma’i “tasty” is grouped with the unaccented noun ame “candy” into a single accented AP.

IP is defined as the prosodic domain within which pitch range is specified: At the beginning of an IP, the speaker chooses a new pitch range that is independent of the former specification. This process is called “pitch reset” (Venditti, 2005). Pitch-range specification of IPs is closely connected with a phonological process called “downstep” by which the pitch peak of each AP is lowered when that AP follows an accented AP. Downstep is displayed in Fig. 2. When multiple accented APs form a single IP, we can observe a staircase-like F0 contour. This is demonstrated in Fig. 3.1

In addition to the progressive lowering of following F0 peaks, downstep may also bring about the raising of preceding peaks. To be more specific, when an IP has many accented APs and therefore a significant amount of downstep can be expected, pitch may be reset to a higher level at the beginning of that IP than would otherwise be the case. This “anticipatory raising,” sometimes claimed to be evidence for preplanning of tone production, is widely documented especially for African languages (e.g., Laniran and Clements, 2003), while it has been only sparsely investigated for Japanese. At least in Fig. 3, the F0 peak of the first AP is notably high arguably because of anticipatory raising brought about by the four downsteps that occur in this IP. If the pitch level at the end of an IP is not raised despite anticipatory raising, then downstep and anticipatory raising result in a large pitch range.

2.BPM¶

The second major element of Japanese intonation is the inventory of BPMs. BPMs are tones that can occur at the end of an AP and contribute to the pragmatic interpretation of the phrase, indicating features such as questioning, emphasis, and continuation (Venditti et al., 2008). Not all APs have BPMs; in fact, most APs are not marked by BPMs. The occurrence of BPMs is not restricted to IP-final or utterance-final APs. They can also occur at the end of IPmedial APs.

The inventory of BPMs indicated in the X-JToBI system is H% (rise), LH% (scooped rise), (HL)% (rise-fall), and HLH% (rise-fall-rise), as well as their variations (Maekawa et al. 2002). Figure 4 depicts these four main types of BPM. As can be seen from the figure, all types of BPM consist of a rise at their beginning. In most cases, the rise starts around the onset of the AP-final mora.

Which pragmatic intentions each of the BPM conveys is not without controversy. Briefly speaking, H% gives prominence to the constituent to which it associates, LH% is generally exploited to signal a question, and HL% is often used in a context where a speaker is explaining some point to a listener (Venditti et al., 2008). HLH% occurs quite infrequently, and, according to Venditti et al. (2008), it gives a wheeling or cajoling quality to the utterance. In Figs. 2 and 3 above, the final morae of the utterances are marked by a LH% (scooped rise) BPM.

注釈

FIG. 1.

Waveforms and F0 contours of unaccented accentual phrase (AP) amai ame “sweet candy” (left) and accented AP uma’i ame “tasty candy.” Vertical lines indicate AP boundaries. The adjective amai “sweet” and the noun ame “candy” in the phrase amai ame “sweet candy” (left) are both lexically specified as unaccented, while the adjective uma’i “tasty” in the phrase uma’i ame “tasty candy” (right) is lexically specified as accented on the second mora/ ma/, which exhibits an F0 fall starting near its end.

注釈

FIG. 2.

Waveform and F0 contour of utterances without and with downstep: an utterance without downstep yubiwa-o wasure-ta onna’-wa dare-desu-ka? “Who is the woman that left the ring behind?” (top), and an utterance with downstep on the third AP yubiwa-o era’nd-ta onna’-wa da’re-desu-ka? “Who is the woman that chose the ring?” (bottom). Dotted vertical lines stand for AP boundaries, and solid vertical lines for IP boundaries. In the utterance in the top panel, an accented AP onna’-wa “woman-TOP” follows unaccented APs yubiwa-o “ring-ACC” and wasureta “left behind,” and thus downstep does not occur. In the utterance in the bottom panel, on the other hand, the accented AP onna’-wa “woman-TOP” follows an accented AP era’nda “chose.” The F0 peak of the AP onna’-wa “woman-TOP” is therefore reduced due to downstep caused by the preceding accented AP. In both utterances, pitch range is reset at the beginning of the last AP da’re-desu-ka “who-COPULA-Q,” whose peak is as high as that in the preceding AP. We can therefore posit an IP boundary between on’nawa and da’re-desu-ka “who-COPULA-Q” in both utterances.

注釈

FIG. 3.

Waveform and F0 contour of an utterance with four successive downsteps, ao’i ya’ne-no ie’-o era’nda onna’-wa da’re-desu-ka? “Who is the woman who chose the house with a blue roof?” Vertical lines indicate AP boundaries and solid vertical lines indicate IP boundaries. Five accented APs, ao’i ya’ne-no ie-o era’nda onna’-wa “blue roof-GEN house-ACC woman-TOP” constitute a single IP. The F0 peaks of the last four APs are iteratively lowered, i.e., downstepped, until pitch reset occurs at the beginning of the following IP. Also, anticipatory raising is observed in the F0 peak of the first AP, which is notably high.

注釈

FIG. 4.

Waveforms and F0 contours of four BPM types—H%, LH%, HL%, and LH%—appearing in the same utterance I’ma-ne… “Just now…”. Squares mark BPMs.

B.Possible impacts of pitch accent and BPM on pitch range¶

There are two elements that can impact pitch ranges in Japanese intonation: Lexical pitch accent (and associated downstep and anticipatory raising) and BPMs. As will be clearer below, each of these two factors functions to expand the pitch range for different reasons. This leads us to hypothesize (1) that the intonational exaggeration in IDS most clearly occurs in those tones that the speaker chooses for pragmatic reasons, which only occur at BPMs in Japanese, (2) that pitch ranges can also be enlarged by pitch accents, whose effect grows as the IPs become longer and the number of accents within the IP increases, and (3) that the interaction of these two factors can superficially mask the intonational exaggeration that may exist in Japanese IDS.

We begin by discussing the first part of this hypothesis. When speakers express pragmatic intent, they may do so by varying the intonation contour. Languages do not, however, typically permit speakers to vary F0 contours without limits—rather, they are restricted by the phonology of the language (cf. Ladd, 1996). In other words, in order to convey specific pragmatic information, speakers are free to choose from an inventory of intonational/pragmatic tones (Ward and Hirschberg, 1985; Pierrehumbert and Hirschberg, 1990), but such pragmatically chosen tones are structured according to the phonology of the particular language. It follows not only that the inventory of these pragmatic tones is languagespecific, but also that the locations in the utterance in which pragmatically chosen tones appear differ across languages. Given that variability in the f0 contour is restricted by the phonology of the language, it is reasonable to assume that intonational exaggeration will be most clearly observed at those locations where the intonation system of a language allows maximum flexibility in varying F0 contours (i.e., the location where pragmatically chosen tones can appear), rather than to assume that it is uniformly distributed over the utterance.

Cross-linguistic differences in the locations of pragmatic tones (as well as in their inventory) may become clear by comparing the intonation system of Japanese to that of English. Figure 5(a) shows the finite-state grammar for English of Pierrehumbert (1980), which can generate all the intonational contours in that language. In this language, all of the tonal categories (pitch accent, phrase accent, and boundary tone) involve a choice of tones. For each type of structure speakers choose one of these alternatives in order to convey different types of pragmatic information (Ward and Hirschberg, 1985; Pierrehumbert and Hirschberg, 1990), and thus these tones are pragmatically chosen ones.

The structure of Japanese intonation (Pierrehumbert and Beckman, 1988; Maekawa et al., 2002; Venditti, 2005) is summarized in the finite-state grammar in Fig. 5(b), which shows that except for the end of the phrase, the F0 contour does not allow much variability. The only tonal options available in this part of the utterance involve the presence or absence of H*þL (a lexical pitch accent). As discussed above, however, this is an intrinsic part of the lexical representation of a word and does not vary according to the speaker’s pragmatic intent.

The contour at the end of the phrase, in contrast, can be much more variable. This is the location allocated to various types of BPM. Unlike lexical pitch accent, the choice of BPMs depends on pragmatic factors. When speakers wish to express some pragmatic information, they have a choice as to whether they assign a BPM to a given phrase and as to what type of BPM they use.

If, as we hypothesized above, intonational exaggeration in IDS should emerge at the pragmatically chosen tones, then in English it should appear anywhere in the contour: It should be observed not only in stressed syllables where pragmatic pitch accents occur but also at the edges of phrases where phrasal accents and boundary tones appear. In Japanese, by contrast, exaggeration should be confined to a more restricted part of the contour, namely, in the BPM at the end of the AP, which is the only location where pragmatically chosen tones are realized in this language.

In Japanese the section of the F0 contour which does not include the BPM, which we will refer to as the BODY (we treat %L, H-, H*þL, and L% collectively as the BODY), is largely determined by the lexical specifications of the words in the phrase and thus any register-induced pitch-range expansion in the BODY is expected to be of minimum magnitude for this language. This prediction is consistent with the findings mentioned above from tone languages such as Chinese and Thai, in which lexically specified tones are densely distributed in utterances and registerinduced pitch-range expansion has been shown to be significantly smaller than that in English (Grieser and Kuhl, 1988; Kitamura et al., 2002; Papousek et al., 1991). Thus, the BODY in Japanese patterns with utterances in tone languages with respect to flexibility in varying F0 contours.

We now turn to the second part of our hypothesis. It concerns the effects of pitch accents and accompanied downstep and anticipatory raising that enlarges the pitch range of the BODY.

One of the effects lexical pitch accent has on pitch range is that it lowers the minimum point of the F0 contour and thus enlarges the overall pitch range of a single AP. As can be seen from Fig. 1, the pitch range of the accented AP uma’i ame (left) is larger than that of an unaccented AP amai ame (right). Downstep is also responsible for the enlargement of the pitch range. Although the primary consequence of downstep is the lowering of the F0 peaks (Hor H*þL) of APs, as shown in Fig. 3, F0 valleys (L%) are also lowered, although to a lesser degree. Because of this lowering of F0 valleys, the pitch range of the IP overall becomes larger every time the accentual fall occurs. Finally, when anticipatory raising occurs in association with downstep, it may also impact the pitch-range expansion. As the number of pitch accents in IP increases, the F0 peak of the initial AP becomes higher and consequently the pitch range of the IP is expected to grow. It is important to note here that pitch-range expansion brought about by factors associated with lexical pitch accent occurs even in the most idealized speech, and therefore intonation is not “exaggerated” here.

注釈

FIG. 5.

Finite-state grammars for English (top) and Japanese (bottom) intonational tunes.

The crucial prediction of this hypothesis is that the effects associated with pitch accents on pitch ranges should be larger in ADS than IDS. It is well documented that utterances in ADS in general contain more words and are longer in duration than those in IDS (Fernald, 1992; Fernald et al., 1989; Grieser and Kuhl, 1988; Newport et al., 1977; Snow, 1977, among others). Assuming that the proportion of accented to unaccented words is not significantly larger in IDS than in ADS (an assumption that is borne out, as discussed in Sec. IV D), ADS utterances, which in general contain more words than IDS utterances, should on average contain more pitch accents than IDS utterances. This leads to the prediction that ADS utterances should have a larger average pitch range of BODY than IDS, if there is no register-induced pitch-range expansion in the IDS utterance. Or, said differently, when we compare utterances in IDS and ADS with the same length, we should find the same pitch range if there is no intonational exaggeration in IDS. If, on the other hand, there is intonational exaggeration, IDS utterances should have larger pitch ranges than ADS utterances of equivalent length.2

Finally, we will discuss the third part of the hypothesis. The pitch-range expansion effect that lexical pitch accents have on the BODY should be larger in the long utterances characteristic of ADS than in the short utterances characteristic of IDS. This effect could thus superficially mask intonational exaggeration that should be observed in BPMs of IDS, if these two phonological entities, pitch accent and BPM, are not taken into consideration. In order to find intonational exaggeration in Japanese, it is therefore necessary first to analyze the BODY and BPMs separately, and second to compare utterances of the same length between ADS and IDS.

III.DATA: JAPANESE IDS CORPUS¶

V.DISCUSSION AND CONCLUSION¶

a IDS was significantly higher (larger) than ADS at less than 0.1% in the post-hoc comparisons of means with Bonferroni correction.

The present study investigated the dynamic aspects of a language’s intonation by examining pitch-range expansion, or intonational exaggeration, in IDS in Japanese, which has been reported to be absent in this language (Fernald et al., 1989). We found that (1) when measured as the difference between maximum and minimum F0 of whole utterances, Japanese IDS showed no pitch-range expansion, replicating the findings of Fernald et al. (1989); (2) while pitch ranges at the locations of BPMs were significantly expanded in IDS, pitch ranges for the rest of the utterance, which the paper calls BODY, were larger in ADS than IDS; (3) the pitch range for BODY is most strongly determined by its length (i.e., the number of pitch accents it contains), and once length is accounted for, the pitch range of the BODY in ADS is in fact no larger than that of IDS. On the basis of these findings, we argue that Japanese IDS does show register-induced pitch-range expansion.

10. Acoust. Soc. Am., Vol. 134, No. 2, August 2013

Igarashi et al.: Exaggerated prosody in infant-directed speech

Max (Hz)

Min (Hz)

Range (st)

(14.26) 183.76 (15.42) (17.66) 170.23 (18.81) (0.54) 3.01 (0.68) (0.69) 5.75 (1.06) (1.24) 9.48 (1.50) (2.09) 11.41 (2.72)

a

n.s.

n.s.

n.s. n.s. n.s. n.s. n.s.

1291

Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 133.9.93.26 On: Fri, 07 Feb 2014 07:08:56

First, we found robust pitch-range expansion at the locations of BPMs, which occur more frequently in IDS than ADS. As discussed in Sec. II, BPMs are tones that are associated with pragmatic interpretations (Venditti et al., 2008), and IDS utterances containing BPMs often involve mothers’ attempt to engage infants by addressing questions to them or seeking agreement by using the sentence-final particle ne (cf. Fernald and Morikawa, 1993). Crucially, when BPMs occur in IDS, each of the tones is produced with an expanded pitch range. This type of pitch-range expansion is likely to be heard as “exaggerated” by listeners and may account for the results of previous studies showing that the intonation of Japanese IDS is perceived to be exaggerated by adults (Horie et al., 2008) and that Japanese IDS is preferred over ADS by infants (Hayashi et al., 2001). Second, as shown in the third analysis (Sec. IV D), the length-induced pitch-range expansion in the ADS BODY is not an “exaggeration” of the intonation. The pitch range of an intonation phrase (IP) is determined primarily by the number of accents the IP contains—the longer the IP (and thus the more accented words it contains), the larger the pitch range. This is independent of register-induced pitchrange modification and occurs regardless of the difference between ADS and IDS. When a speaker produces a long utterance with a pitch range that is normal for its length, she/he is under no pressure to “exaggerate” the intonation, nor is it likely to be heard as “exaggerated” by a listener. In fact, when ADS and IDS utterances of equal length were compared, there was no difference in pitch ranges between the two registers. These results highlight the usefulness of an intonational phonological framework to describe how intonation is modified in a specialized speech register. Our analysis has shown that pitch-range expansion in Japanese IDS has previously been overlooked, because no reference has been made in the prior scholarly literature to phonological events specific to Japanese—specifically, BPM and lexical pitch accents, as well as utterance length and the interactions between these factors. We do not mean to argue that pitch-range expansion in IDS is not universal. On the contrary, our findings provide additional support to the view that it is. Our findings are novel in that they successfully demonstrate that not all phonological tones are subject to the paralinguistic modification characteristic of a specialized speech register. Specifically, our analysis suggested that pitch-range expansions in IDS are not realized in the same way in every language, but are instead implemented within a languagespecific system of intonation. When there is a desire or pressure to exaggerate the intonation, speakers seem to do so by expanding the pitch range at the location where flexibility in varying contours is most tolerated. In phonological terms, this is the location where pragmatically chosen tones are realized. In the case of Japanese, these are BPMs at the boundaries of prosodic phrases, while in the case of English, they are not only phrase accents and boundary tones at the phrasal boundaries, but pitch accents at the locations of stressed syllables. It has been commonly assumed that paralinguistic pitch-range modifications (which should include intonational exaggeration in IDS) can occur globally 1292

10. Acoust. Soc. Am., Vol. 134, No. 2, August 2013

irrespective of what tones are present in the utterance [cf. Ladd (1996), Chap. 7]. The present results, however, showed that only certain tones can undergo paralinguistic modifications. Our study, therefore, promises to shed light on the phonetics of pitch range variation. One implication of the present study is that crosslinguistic differences in IDS intonation may be better captured by re-examining them with reference to the intonation system of each language. The intonational exaggeration in Japanese is camouflaged by the pitch range of BODY that increases linearly as the number of pitch accents in an IP increases. In English, by contrast, such an increase is not expected, and thus the register-induced exaggeration can be captured straightforwardly by the conventional, phonologically uninformed, method of measuring the pitch range—the maximum minus the minimum F0 of an entire utterance. The same method, however, is not sufficient to capture the two competing forms of pitch-range expansion in Japanese. This leads us to speculate that the magnitude of intonational exaggeration in some tone languages (Thai and Chinese) is in fact larger than what has previously been reported. In these languages, the pitch range in IDS are generally reported to be smaller (e.g., 6 to 7 semitones; Grieser and Kuhl, 1988; Kitamura et al., 2002; Papousek et al., 1991) than in English and other Germanic and Romance languages (10 to 12 semitones; Fernald et al., 1989; Kitamura et al., 2002). Examining the intonation of these languages with reference to their intonation systems may allow us to better understand the specific ways these languages modulate their intonation. It might provide a clue as to why the intonation of Mayan-Quiche speaking mothers do not show the typical pitch characteristics of IDS (Bernstein Ratner and Pye, 1984; Ingram, 1995). At the same time, our data robustly showed that Japanese IDS is produced with higher pitch than ADS. This is consistent with many previous studies of English and other Germanic/Romance languages (Fernald, 1989; Fernald et al., 1989), Japanese (Fernald et al., 1989) as well as tone languages (Papousek et al., 1991; Liu et al., 2007; Kitamura et al., 2002). It has sometimes been argued that the speech modulation seen in IDS may be driven by biological factors common across species, and that both adults’ tendency to produce higher-pitched speech when addressing infants and infants’ preference for higher-pitched voices may be part of a biologically driven phenomenon (Morton, 1977; Trainer and Zacharias, 1998). Thus, the results of the present study help show that there is a complex interplay of universal and language-specific factors that contribute to the pitch modulation that is characteristic of IDS intonation.

ACKNOWLEDGMENTS¶

We thank the children and mothers for their participation in the recordings of IDS and ADS used in the present study. We also thank Akira Utsugi, Kikuo Maekawa, and Andrew Martin for their helpful comments. The study reported in this paper was supported in part by a Japanese government Grant-in-Aid for Young Scientists B #23720207 to Y.I. and a Grant-in-Aid for Scientific Research A#21610028 to R.M. Igarashi et al.: Exaggerated prosody in infant-directed speech

Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 133.9.93.26 On: Fri, 07 Feb 2014 07:08:56

• Bernstein Ratner, N., and Pye, C. (1984). “Higher pitch in BT is not universal: Acoustic evidence from Quiche Mayan,” J. Child Lang. 11(3), 515–522.

Boersma, P., and Weenink, D. (2006). “Praat, a system for doing phonetics by computer,” Glot International, Vol. 5, pp. 341–345. Bornstein, M. H., Tamis-LeMonda. C. S., Tal, J., Ludemann, P., Toda, S., Rahn, C. W., P^echeux, M. G., Azuma, H., and Vardi, D. (1992). “Maternal responsiveness to infants in three societies: the United States, France, and Japan,” Child Dev. 63(4), 808–821. Cooper, R. P., Abraham, J., Berman, S., and Staska, M. (1997). “The development of infants’ preference for motherese,” Infant Behav. Dev. 20(4), 477–488. Crystal, D. (2008). A Dictionary of Linguistics and Phonetics, 6th ed. (Blackwell Publishing, Malden, MA). Ferguson, C. A. (1977). “Baby talk as a simplified register,” in Talking to Children: Language Input and Acquisition, edited by C. E. Snow and C. A. Ferguson (Cambridge University Press, London), pp. 209–235. Fernald, A. (1989). “Intonation and communicative intent in mothers’ speech to infants: Is the melody the message?,” Child Dev. 60, 1497–1510. Fernald, A. (1992). “Meaningful melodies in mothers’ speech to infants,” in Nonverbal Vocal Communication: Comparative and Developmental Approaches, edited by H. Papousek, U. Jurgens, and M. Papousek (Cambridge University Press, Cambridge), pp. 262–282. Fernald, A. (1993). “Approval and disapproval: Infant responsiveness to vocal affect in familiar and unfamiliar languages,” Child Dev. 64, 657–674. Fernald, A., and Kuhl, P. K. (1987). “Acoustic determinants of infant preference for motherese speech,” Infant Behav. Develop. 10, 279–293. Fernald, A., and Morikawa, H. (1993). “Common themes and cultural variations in Japanese and American mothers’ speech to infants,” Child Dev. 64, 637–656. Fernald, A., Taeschner, T., Dunn, J., Papousek, M., de Boysson-Bardies, B., and Fukui, I. (1989). “A cross-language study of prosodic modifications in mothers’ and fathers’ speech to preverbal infants,” J. Child Lang. 16, 477–501. Grieser, D. L., and Kuhl, P. K. (1988). “Maternal speech to infants in a tonal language: Support for universal prosodic features in motherese,” Dev. Psychol. 24(1), 14–20. Gussenhoven, C. (2004). The Phonology of Tone and Intonation (Cambridge University Press, Cambridge). Hayashi, A. (2004). “Zen-gengoki no onsei chikaku hattatsu to gengoshuutoku ni kansuru jikken-teki kenkyuu” (“An experimental study of preverbal infants’ speech perception and language acquisition”), Doctoral Dissertation, Tokyo Gakugei University, Tokyo, Japan. Hayashi, A., Tameka, Y., and Kiritani, S. (2001) “Developmental change in auditory preferences for speech stimuli in Japanese infants,” J. Speech Lang. Hear. Res. 44, 1189–1200. Horie, R., Hayashi, A., Shirasawa, K., and Mazuka, R. (2008). “Mother, I don’t really like the high-pitched, slow speech of Motherese: Crosslinguistic differences in infants’ reliance on different acoustic cues in infant directed speech,” XVIth Biennial International Conference on Infant Studies, Vancouver, Canada. Ingram, D. (1995). “The cultural basis of prosodic modifications to infants and children: A response to Fernald’s universalist theory,” J. Child Lang. 22(1), 223–233. Inoue, T., Nakagawa, R., Kondou, M., Koga, T., and Shinohara, K. (2011).“Discrimination between mothers’ infant- and adult-directed speech using hidden Markov models,” Neurosci. Res. 70, 62–70. Jun, S.-A. (2005). Prosodic Typology: The Phonology of Intonation and Phrasing (Oxford University Press, New York). Kitamura, C., Thanavishuth, C., Burnham, D., and Luksaneeyanawin, S. (2002). “Universality and specificity in infant-directed speech: Pitch modifications as a function of infant age and sex in a tonal and non-tonal language,” Infant Behav. Dev. 24, 372–392. Kubozono, H. (1993). The Organization of Japanese Prosody (Kurosio Publishers, Tokyo). Ladd, D. R. (1996). Intonational Phonology (Cambridge University Press, Cambridge). Laniran, Y. O., and Clements, G. N. (2003). “Downstep and high raising:Interacting factors in Yoruba tone production,” J. Phonetics 31, 203–250. Liu, H.-M., Tsao, F.-M., and Kuhl, P. K. (2007). “Acoustic analysis of lexical tone in Mandarin infant-directed speech,” Dev. Psychol. 43(4),912–917. Maekawa, K. (2003). “Corpus of Spontaneous Japanese: Its design and evaluation,” in Proceedings of ISCA and IEEE Workshop on Spontaneous Speech Processing and Recognition, Tokyo, pp. 7–12. Maekawa, K., Kikuchi, H., Igarashi, Y., and Venditti, J. (2002). “X-JToBI:An extended J_ToBI for spontaneous speech,” in Proceedings of the 7th International Conference on Spoken Language Processing, Denver, CO, pp. 1545–1548. Mazuka, R., Igarashi, Y., and Nishikawa, K. (2006). “Input for learning Japanese: RIKEN Japanese Mother-Infant Conversation Corpus,” Technical report of IEICE, TL2006-16 106(165), pp. 11–15. McCann, J., and Peppe, S. (2003). “Prosody in autism spectrum disorders: A critical review,” Int. J. Lang. Commun. Disord. 38, 325–350. Morton, E. S. (1977). “One the occurrence and significance of motivationstructural rules in some bird and mammal sounds,” Am. Nat. 111(981), 855–869. Newman, R. S., and Hussain, I. (2006). “Changes in preference for infantdirected speech in low and moderate noise by 5- to 13-month-olds,” Infancy 10(1), 61–76. Newport, E., Gleitman, H., and Gleitman, L. (1977) “Mother, I’d rather do it myself: Some effects and non-effects of maternal speech style,” in Talking to Children: Language Input and Acquisition, edited by C. E. Snow and C. A. Ferguson (Cambridge University Press, London), pp. 109–150. Papousek, M., Papousek, H., and Symmes, D. (1991). “The meanings of melodies in motherese in tone and stress languages,” Infant Behav. Dev. 14(4), 415–440. Pegg, J. E., Werker, J. F., and McLeod, P. J. (1992). “Preference for infantdirected over adult-directed speech: Evidence from 7-week-old infants,” Infant Behav. Dev. 15, 325–345. Pierrehumbert, J. (1980). “The phonology and phonetics of English intonation,” Ph.D. dissertation, MIT. Pierrehumbert, J., and Beckman, M. E. (1988). Japanese Tone Structure (MIT Press, Cambridge). Pierrehumbert, J., and Hirschberg J. (1990). “The meaning of intonational contours in the interpretation of discourse,” in Intentions in

10. Acoust. Soc. Am., Vol. 134, No. 2, August 2013

Igarashi et al.: Exaggerated prosody in infant-directed speech

1

Various linguistic factors bring about pitch reset at the IP boundary; these include syntactic constituency and focus (Pierrehumbert and Beckman, 1988; Selkirk and Tateishi, 1991; Kubozono, 1993). In the case of the utterances in Figs. 2 and 3, pitch reset was induced by the focus indicated by wh-element da’re “who.” 2 We do not mean to argue that the pitch range of English (as well as arguably those of many other languages) is not influenced by downstep and anticipatory raising. English utterances can have staircase-like contours, such as one having multiple H*þL pitch accents. The contour of such an utterance would resemble the Japanese one in Fig. 3. In this case, a large pitch range would be expected due to the progressive lowering of the F0 bottoms and the anticipatory raising of the peaks. The difference is that these contours constitute only a small part of the large inventory of contours in English, and the majority of utterances, which have other types of contours, are not subject to downstep or downstep-induced pitch-range expansion. Consequently, simply because utterances are longer in ADS in English and other languages, their pitch ranges are not expected to increase to the same degree as those in Japanese. 3 See Supplementary Material 1 at [http://lang-dev-lab.brain.riken.jp/igarashi-jasa-suppl.html], for inter-labeler reliability with X-JToBI. 4 See Supplementary Material 2 at [http://lang-dev-lab.brain.riken.jp/igarashi-jasa-suppl.html] for the frequency each BPM type. 5 See Supplementary Material 3 at [http://lang-dev-lab.brain.riken.jp/igarashi-jasa-suppl.html] for post-hoc comparisons of ANOVAs. 6 See Supplementary Material 4 at [http://lang-dev-lab.brain.riken.jp/igarashi-jasa-suppl.html] for duration of morae which bear BPMs. 7 We also analyzed the pitch characteristics of the BODY regardless of its length. The results revealed that although the pitch of IDS is higher than ADS, the pitch ranges of the BODY are larger in ADS than in IDS. See Supplementary Material 5 at http://lang-dev-lab.brain.riken.jp/igarashijasa-suppl.html for details. The larger pitch range in ADS is accounted for by the effect of the length of the utterances, as shown in Sec. IV D. 8 See Supplementary Material 5 at [http://lang-dev-lab.brain.riken.jp/igarashi-jasa-suppl.html] for analyses of the average length of Utterances and Intonation Phrases based on several measures. 9 See Supplementary Material 6 at [http://lang-dev-lab.brain.riken.jp/igarashi-jasa-suppl.html] for proportion of accented versus unaccented words.

1293

Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 133.9.93.26 On: Fri, 07 Feb 2014 07:08:56

Communication, edited by P. Cohen, J. Morgan, and M. Pollack (MIT Press, Cambridge, MA), pp. 271–311. Selkirk, E. O., and Tateishi, K. (1991). “Syntax and downstep in Japanese,” in Interdisciplinary Approaches to Language: Essays in Honor of S.-Y. Kuroda, edited by C. Georgopoulos and R. Ishihara (Kluwer Academic, Dordrecht), pp. 519–543. Snow, C. E. (1977). “Mothers’ speech research: From input to interaction,” in Talking to Children: Language Input and Acquisition, edited by C. E. Snow and C. A. Ferguson (Cambridge University Press, London), pp. 31–49. Soderstrom, M. (2007). “Beyond babytalk: Re-evaluating the nature and content of speech input to preverbal infants,” Dev. Rev. 27, 501–532. Toda, S., Fogel, A., and Kawai, M. (1990). “Maternal speech to three-monthold infants in the United States and Japan,” J. Child Lang. 17(2), 279–294.

1294

10. Acoust. Soc. Am., Vol. 134, No. 2, August 2013

Trainer, L. J., and Zacharias, C. A. (1998). “Infants prefer higher-pitched singing,” Infant Behav. Dev. 21(4), 799–806. Venditti, J. (2005). “The J_ToBI model of Japanese intonation,” in Prosodic Typology: The Phonology of Intonation and Phrasing, edited by S. A. Jun (Oxford University Press, New York), pp. 172–200. Venditti, J., Maekawa, K., and Beckman, M. E. (2008). “Prominence marking in the Japanese intonation system,” in Handbook of Japanese Linguistics, edited by S. Miyagawa and M. Saito (Oxford University Press, New York), pp. 456–512. Ward, G., and Hirschberg, J. (1985). “Implicating uncertainty: The pragmatics of fall-rise,” Language 61, 747–776. Williams, C. E., and Stevens, K. N. (1972). “Emotions and Speech: Some acoustical correlates,” J. Acoust. Soc. Am. 52, 1238–1250.

Igarashi et al.: Exaggerated prosody in infant-directed speech

Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 133.9.93.26 On: Fri, 07 Feb 2014 07:08:56

1.Introduction¶

Restricted repetitive behavior has been a defining feature of the Autism Spectrum Disorders (ASD) since the original description of autism (Kanner, 1943), and by diagnostic convention, all individuals with ASD display some form of these “restricted repetitive and stereotyped patterns of behavior, interests, and activities” (Diagnostic and Statistical Manual for Mental Disorders-Forth Edition [DSM-IV], American Psychiatric, [APA], 1994:71). Although ASD is associated with a wide range of specific forms of atypical repetition, this issue received far less research attention than social and communication deficits. Indeed, it was not our original attention to examine the prosody of ASD high functioning (ASD-HF) children from the perspective of the presence or the absence of repetitive behavior, we were concentrating on “prosody” within the context of linguistic behavior - whether or not the manifestation of the “different” prosody by ASD-HF individuals may reflect “delays and deficits in language and communication”, which is another core feature of ASD. However, the data we collected in our research brought this issue into focus and raised new questions regarding the centrality of the restricted repetitive behaviors in ASD.

This chapter is based on results and insights from linguistic research. This research (Green, 2010) comparing and contrasting the prosodic features of 20 peer-matched 9-13 year old male Israeli Hebrew-speaking participants (10 ASD-HF subjects and 10 controls without developmental disorders (WDD) strongly indicated that the prosodic features that were examined exhibited a limited and repetitive repertoire in the ASD-HF population compared with the prosodic features of the WDD control population (Green, 2005; Green & Tobin 2008a, b, 2009 a, b, c; Green, 2010). Furthermore, this significant limited repetitive repertoire of behavior patterns was also exhibited in the extra-linguistic and the linguistic (lexical) domains of the ASD-HF participants.

2.The experimental research¶

As already noted, this chapter is based on experimental research and deals with the “restricted repetitive behavior” phenomenon. In the original linguistic-oriented research there were four major goals:

1. To describe, compare and contrast the phonetic realization of the fundamental frequency and the prosodic features of intonation in the language of children with ASD- HF and WDD children,

2. To establish a methodology which allows the analysis of more than one feature of prosody simultaneously,

3. To make use of instrumental measurements, (using recently developed speech technology tools) as well as perceptual analysis, and

4. To explain the results within the context of the theory of Phonology as Human Behavior (PHB) (e.g. Diver, 1979, 1995; Tobin, 1997, 2009), a linguistic theory which declares that:

   1. Language is a symbolic tool, whose structure is shaped both by its communicative function and the characteristics of its users (Tobin, 1990, 1993, 1994, 2009), and
   2. Language represents a compromise in the struggle to achieve maximum communication using minimal effort as presented in the theory of Phonology as Human Behavior (PHB) (Diver, 1979,1995; Tobin, 1997, 2009).

Our empirical data were drawn from the speech samples of 20 children between the ages 9- 13 years, in two main groups:

1. Research group: subjects diagnosed clinically with ASD-HF (N=10)
2. Control group: participants without developmental disorders (WDD, N=10)

The research group includes ten children with ASD aged 9-13 years. They were recruited from mainstream schools that have special education classes for children with ASD. The ASD diagnosis was made by a child psychiatrist who determined that the child met the DSM-IV, APA (1994) criteria for autism. Each child’s special needs were discussed and defined by an “Evaluation Committee”, entrusted with the placement of special needs pupils in appropriate class settings. For all of the children in this group the committee determined that a special class for children within the ASD spectrum is required. IQ scores were re- assessed by the school psychologist within the current year using the Wechsler Intelligence Scale for Children - a Revised Edition [WISC-R]. For the purposes of this research, High Functioning is defined by an IQ 85 and above. All ASD subject‘s have typical, within the average, school performance in the mainstream class in language and reading, as reported by their teachers.

The control group was composed of children without developmental disorders (WDD) and was drawn from the same schools as the research group. Similar to the research group subjects, in their teachers’ judgment, all the children are average students and do not exhibit any particular academic difficulties or exceptional abilities. The group members have not been tested to determine their IQ scores, but from the information received in interviews with the teachers and parents it can be assumed that they have intelligence in normal range. Their parents report that they have not been referred to a specialist for any developmental reasons. In our study, two language measures were used for peer matching. In addition to similar chronological ages (within two month), the peers were matched on the basis of (a) language fluency in spontaneous-speech sample as measured in MLU-W (within the minimal linguistic unit of one word) and (b) the standardized score of the verbal part in the IQ test within the norm or above. Match between subjects and controls are presented in Table 1.

The analysis of the speech samples of this group provides the basis for the characterization of the prosodic features of Israeli Hebrew (Green, 2009a; Green & Tobin 2009b, c; Green, 2010). All participants were male and at least second generation Israeli-born, and were monolingual speakers of Israeli Hebrew (IH). All participants were from comparable socioeconomic backgrounds and attend mainstream schools. Their mothers all have at least 12 years of education, an indication of socioeconomic status, since maternal education level is the most significant predictor of language functioning in children (Dollaghan et al., 1999). None of the members of the participants’ immediate families has learning or other known disabilities.

ASD-HF subjects WDD control group

Research Control Group MLU Group MLU- ASD-HF Mo VIQ PIQ Age -W WDD Mo Age W 1-ADR 13 109 94 9:0 5.62 11-AVY 13 9:0 5.28 2-ITE 17 121 91 9:2 5.34 12-NIS 15 9:3 4.04 3-UDX 15 108 94 10:8 7.86 13-IDR 17 10:7 6.94 4-YOL 17 111 101 11:1 5.77 14-YVO 16 10:11 5.16 5-RAE 12 86 97 11:6 5.68 15-ITS 17 11:6 4.04 6-BAB 17 109 97 11:11 7.42 16-AVS 16 11:9 7.18 7-ETR 16 100 102 12:5 5.14 17-LIS 13 12:3 5.22 8-TOB 15 108 99 12:8 6.42 18-IDW 16 12:6 6.20 9-NOR 14 90 99 13:0 6.5 19-OMX 14 12:11 5.86 10-OMG(*) 14 89 85 13:0 4.8 20-IDS 17 12:11 6.1

Mo=Mother’s years of Education, VIQ/PIQ= verbal/performance Intelligence score (WISC-R) (*) Participants 10 and 20 do not meet the requirements of the definitions used for peer matching, and are consequently excluded from comparison between the groups. Their results are, however, included when the discussion is about differences within the group.

Table 1. Matched peers and subject’s characteristics

The speech samples were collected at the participant’s house, in his own room. There were three types of elicitation tasks: (a) Repetition: this task comprised four sentence pairs, a WH- Question and its answer, (b) Reading Aloud: participants were asked to read a short story, and (c) Spontaneous speech: these were elicited spontaneous speech sequences in response to open questions, relevant to the child’s daily life. In order to conduct acoustic analyses the speech files were digitized at a rate of 44.1KHz with 16-bit resolution, directly into a laptop computer (Hp Compaq 6710b), using the speech- recording software Audacity (a software package for recording and editing sound files) and a small microphone. The data were subsequently analyzed using the speech analysis program Pratt, version 5.0.30 (Computer program, from http://www.praat.org/). Scripts were written to extract data from the transcriptions. Script is a short program that is used to automate Pratt activities and enables the analysis of large data sets, quick processing of information and results, preparation for the use of simple statistics tools, and generation of summary information for control purposes, i.e. to identify errors in the manual transcription process.

3.Restricted repetitive behavior¶

Restricted repetitive behaviors are a heterogeneous group of behaviors and a wide range of specific forms of atypical repetition that have been identified and described with relation to ASD (e.g. APA, 1994; Bodfish et al., 2000; Esbensen et al., 2009; Kanner 1943; Lewis & Bodfish, 1998; Militerni et al., 2002; Richler et al., 2007; Rutter, 1996; Szatmari et al., 2006; Turner, 1999). This restricted repetitive behavior can be observed across individuals with ASD, and multiple categories of abnormal repetition can occur within the individual with autism (e.g. Lewis & Bodfish, 1998; Wing & Gould, 1979). These behaviors can be socially inappropriate, increase the plausibility of living in a more restricted environment, and stigmatizing (Bonadonna, 1981; Durand & Carr 1987, Varni et al., 1979).

Several researchers who examined age related aspects of repetitive behavior patterns in ASD suggested that age and level of functioning are associated with variation in the manifestation of restricted repetitive behaviors in individuals with ASD (e.g. Esbensen et al., 2009; Militerni et al., 2002; Lam & Aman, 2007). The overall severity of the ASD has been shown to be significantly positively correlated with the overall severity of repetitive behaviors (e.g. Campbell et al., 1990; Prior & MacMillan, 1973). Esbensen et al. (2009) examined the restricted repetitive behaviors among a large group of children and adults with ASD in order to describe age related patterns of symptom expression and examine if age related patterns are different for the various types of restricted repetitive behaviors. In this research, they combined data from several previous studies to have a large sample size (n = 712), spanning a broad age range (age 2–62), and they measured restricted repetitive behaviors using a single instrument, the Repetitive Behavior Scale-Revised (RBS-R: Bodfish et al., 2000) with the modification of the subscales (Lam & Aman, 2007). The empirically derived subscales include: Stereotyped Behavior (movements with no obvious purpose that are repeated in a similar manner), Self-injurious Behavior (actions that cause or have the potential to cause redness, bruising, or other injury to the body), Compulsive Behavior (behavior that is repeated and performed according to a rule or involves things being done ‘‘just so’’), Ritualistic/sameness Behavior (performing activities of daily living in a similar manner; resistance to change, insisting that things stay the same), and Restricted Interests (limited range of focus, interest, or activity). Their analyses suggest that repetitive behaviors are less frequent and less severe among older individuals than among younger individuals regardless of whether examining total display of restricted repetitive behaviors, or whether examining each of the various subtypes. One may ask whether restricted repetitive behaviors decrease with age or whether they merely take a different form. A thought previously arise by Piven et al. (1996). Piven’s idea was that manifestation of ASD changes as the individual develops.

Other research has suggested that the expression of restricted repetitive behaviors may be influenced by level of functioning (e.g. Bartak & Rutter, 1976; Campbell et al., 1990; Gabriels et al., 2005; Le Couteur et al., 2003; Turner, 1999). Low IQ or presence of mental retardation has been shown to be associated with increased occurrence of repetitive behaviors in autism including stereotypy and self-injury (Bartak & Rutter 1976; Campbell et al., 1990). Turner (1997) proposed a taxonomy of repetitive behavior; consisting of eleven categories and in a later review (Turner, 1999) suggested that human repetitive behaviors can be divided into (a) lower-level and (b) higher-level categories. Lower-level repetitive behaviors include dyskinesia (involuntary, repetitive movements), tics, repetitive manipulation of objects, repetitive forms of self-injurious behavior and stereotyped movements. Turner’s review indicates that although some stereotyped movements and repetitive manipulation of objects might be differentiating features of autism, there are some lower-level repetitive behaviors that may rather be related to ability level or the presence of organic pathology (e.g. Bishop et al., 2006; Bodfish et al., 2000; Cuccaro et al., 2003; Esbensen et al., 2009; Fecteau et al., 2003; Militerni et al., 2002; Lam & Aman, 2007; Szatmari et al., 2006). Irrespective of whether these low-level repetitive behavioral characteristics are unique to ASD or exist in a wider range of organic pathological conditions, they are all repetitive extra-linguistic behaviors.

The high-level repetitive behaviors include circumscribed interests, attachments to objects, insistence on maintenance of sameness and repetitive language. Turner (1999) suggested that certain types of higher-level behavior may be characteristic of and restricted to individuals with ASD once a certain level of development has been achieved.

4.References¶

Abstract¶

本稿は自発音声における breth groupの言語学的構造と呼吸の運動学の関係を調査した. 26名のドイツ人女性話者のPlethysmograph [1] の磁場の平均値を記録した. brearh group は単一呼気における発話のインターバルと定義した. それぞれのグループ間で,言語学的パラメータ(句のタイプ数や,シラブル数,ためらいなど句のタイプ数や,シラブル数,言いよどみ等)を計測し,吸気と関連づけた. 呼吸グループの平均持続時間は3.5 sec 以上であった. 呼吸ブループのほとんどが1ー3の句 [2] を構成していた. 53%以上は句マトリックスの冒頭で開始しており,24%が句の中に埋め込まれており,23%は不完全な句(継続,反復,言いよどみ)とともに現れた. 吸音の深さと長さは,はじめの句のタイプの関数として,呼吸グループの長さを反映して変化し, 発話計画と呼吸の制御の間にいくつかの交互作用が現れた. これらの結果は,自発的な発話での音声·計画と呼吸制御の相互作用をより良く理解するために有益である, 調査結果はスピーチセラピーやその技術の応用に関連している.

This paper investigates the relation between the linguistic structure of the breath group and breathing kinematics in spontaneous speech. 26 female speakers of German were recorded by means of an Inductance Plethysmograph. The breath group was defined as the interval of speech produced on a single exhalation. For each group several linguistic parameters (number and type of clauses, number of syllables, hesitations) were measured and the associated inhalation was characterized. The average duration of the breath group was ~3.5 s. Most of the breath groups consisted of 1-3 clauses; ~53% started with a matrix clause; ~24% with an embedded clause and ~23% with an incomplete clause (continuation, repetition, hesitation). The inhalation depth and duration varied as a function of the first clause type and with respect to the breath group length, showing some interplay between speech-planning and breathing control. Vocalized hesitations were speaker-specific and came with deeper inhalation. These results are informative for a better understanding of the interplay of speech-planning and breathing control in spontaneous speech. The findings are also relevant for applications in speech therapies and technologies.

訳者注

1. Introduction¶

数秒のタイムスケールにおいて,音声発話とは発声を伴う長い呼気に付随する短い吸気の休止の連続である. 一つの呼気における音声発話のインターバルは一般的には ” breath group ” として定義されている. これは言語学的,コニュニケーション的,生理的な制約に依存している. Breath group は韻律や発話の知覚に対する重要なユニットでもある[1]. 本稿ではドイツ語の breath group を以下の2つの問いに注目して解析をした.

1. Brearh group の言語学的な構造は何なのか?
2. 吸気をしている間にこの構造を予測しているのか?

On a time-scale of several seconds, speech production is a sequence of short inhalations pauses followed by long exhalations with phonation. The interval of speech produced on a single exhalation is commonly defined as the breath group. It relies on linguistic, communicative and physiological constraints. The breath group is also an important unit for prosody and speech perception [1]. The present paper analyses the breath group in German spontaneous speech with respect to two main questions:

1. What is the linguistic structure of the breath group?
2. Is this structure anticipated during inhalation?

吸気の深さと続く breath group の言語学的構造体の長さの関係は発話計画と換気の相互作用を反映している[2-8]. これらの関係は読み上げでも,自発音声でも調査されてきた. これらの研究は異なる発話タスクを含んでおり(例えば,センテンスやテキストの読み上げ,異なる認知負荷での自発音声など),異なる方法で呼吸のパラメータを推定している(呼吸音ノイズの検知,例えば[2,9], 口や鼻からの風の計測[3-8,11-16],そして音響と運動学的方法の組み合わせ[17]).

The relation of the inhalation depth and duration to the linguistic structure of the upcoming breath group reflects the interplay of speech-planning with ventilation [2-8]. These relations have been investigated in both read and spontaneous speech. These studies involved different speech tasks (e.g. sentences and texts reading, spontaneous speech with different cognitive load) and estimated breathing parameters with different methods (detection of breath noises, e.g. [2,9]; measurement of the air flow from the mouth and nose, e.g.[10]; monitoring of the kinematics of the chest wall, e.g. [3-8, 11-16], see also [17] for a comparison between acoustic and kinematic methods).

いくつかの研究ではセンテンスや文章において吸気時の breath group の長さの予測を示している. 吸気の深さと長さはセンテンスの長さと共に増加する[6,11-14,16]. 加えて,文読み上げの際の吸気は次に来る breath group の文法的な複雑さ(句の数)とは明確な関係はない. テキストリーディングでは,ほぼ100%の吸気パルスが句読点や接続語(例えば, and など)によってしめされる文法的な境界において生じる. これらは brearh group がセマンティックな構造をとっていることを示す結果である[2-6, 8-10, 15]. テキストリーディングにおいては,吸気の深さも長さも構文上のマークに対しては異なっている(例えば 段落 > ピリオド > コンマ)[5-6].

Several studies show an anticipation of the breath group length during the preceding inhalation for sentence and text reading. The inhalation depth and duration increase with the sentence length [6, 11-14, 16]. Furthermore, inhalations in sentence reading are not clearly related to the syntactic complexity (number of clauses) of the upcoming breath group [12-13]. In text reading, almost 100 % of the inhalation pauses occurs at syntactic boundaries, indicated by punctuation marks or conjunctions (e.g. and). These results show that the breath groups are syntactically structured [2-6, 8-10, 15]. In text reading, the inhalation depth and duration also differed with respect to syntactic marks (e.g. paragraph > period > comma) [5-6].

自発音声においては,呼吸パレスはシンタックスのみではなく,言語学的なコンテンツを生成するために必要な認知プロセスによっても支配されている[2,4,7-8,15]. このプロセスは発話のフローの言いよどみを導入する. 自発音声においては約80%の呼吸パレスが文法的な成分において発声する. 吸音の平均的なアンプリチュードと持続時間はテキスト読み上げに似ており,次にくる brearh group の長さを反映している. brearh group の平均持続時間は読み上げ音声よりも長い[4,7,15,18-19を参照]. これらのパラメータのばらつきの範囲はテキスト読み上げと比較して自発音声のほうが大きい. 自発音声は呼吸に関連しており,異なる機能を有すると想定される有声のためらい発話(/うー/とか/うーむ/とか)によって特徴付けられる[20,21].

In spontaneous speech, the breathing pauses are not only governed by syntax but also by the cognitive processing required to generate the linguistics content [2,4,7-8,15]. This process introduces disfluencies in the speech flow. In spontaneous speech about 80% of the breathing pauses occur at syntactic constituents; the average amplitude and duration of inhalation are similar to text reading and are reflecting the length of the upcoming breath group. The average duration of breath groups is also longer than in text reading [see: 4, 7, 15, 18-19]. The ranges of variability of these parameters are larger in spontaneous speech as compared to text reading. Spontaneous speech is also characterized by the production of vocalized hesitations (uh, um) that have been assumed to have different functions and have been related to breathing [20, 21].

本稿では,ドイツ語の自発音声における, 呼吸の運動学と brearh group 言語学的な構造との関係を評価した. 先行研究に習って,我々は brearh group の中の文法的な構造(句の数)とシラブルの数を考慮した. brearh group 内の句の種類(マトリックス句または埋め込み句)と言いよどみ(ためらい - うーん,うーむ,繰り返し)も調査した. brearh group における最初の句の種類(マトリックス句か,埋め込み句か)は言語学的構造と関連した吸気の位置の指標である. 言いよどみ,特に有声のためらいと呼吸との関連は発話計画を含む認知プロセスに付いて有益な情報を持っている.

This paper evaluates the relationship between the kinematics of breathing and the linguistic structure of the breath group in German spontaneous speech. As in previous studies we consider the syntactic structure (number of clauses) and the number of syllables in the breath group. We also analyzed the type of clauses (matrix, embedded clause) and disfluencies (hesitations – uh, um, repetition, repairs...) in the breath group. The type of the first clause (matrix clause or embedded clause) in the breath group is an indicator of the location of inhalation relative to the linguistic structure. The association of breathing to disfluencies, and especially vocalized hesitations, is informative about the cognitive process involved in speech planning.

2. Experiment¶

2.2. Experimental settings and procedure¶

参加者は,指向性マイクと2つのスピーカーの正面に立った(Figure1.A). 自発音声タスクが大きな実験プロトコルの一部である. 休憩時,及び,短い音読時の呼吸を少し収録したあと,参加者はドイツ語の男性,あるいは,女性のネイティブスピーカーによって読み上げられた,10個の簡単な文章の録音音声をよく聞くように指示された. 各トラックはスピーカーを通して再生された. これらの文章を聞いたあと,参加者は内容の簡単な要約を行った. 呼吸運動の観察を妨げるような可能性のある動きを制限するため, 参加者はトランクに沿って,手を維持するように指示された. 肺活量(VC)操作は,胸部の変位とVCによって誘導された腹部を推定するための手順の最後に実行された. そのため,被験者は,できる限り多くの空気を吐き出し,できる限り多くの空気を吸入する.

Participants were standing up in front of a directional microphone and two loudspeakers (Figure 1.A). The spontaneous speech task was part of larger experimental protocol. After a short recording of breathing at rest and short reading, participants were instructed to listen attentively to the audio recordings of ten brief texts (151±22.1 syllables), read by a male or a female native speaker of German. The tracks were played back through the loudspeakers. After listening to each text, participants briefly summarized the story. In order to limit the movements that could interfere with the monitoring of breathing kinematics, participants were instructed to keep their hands along their trunk. Vital capacity (VC) maneuvers were run at the end of the procedure to estimate the displacement of the rib cage and the abdomen induced by VC. To do so, subjects exhaled as much air as they could and then inhaled as much air as they could.

Figure 1: - (A) 実験セットアップ - (B) 吸気フェーズ(I)と呼気フェーズ(E)の呼吸運動の例. - (C) breath groupsのラベリングとシラブル,句数.

• Hはためらい発話部分を示している.詳細は本文参照.

3. Results¶

Table I. Description of the breath groups according to the number of clauses and to the type of the first clause. NB: Number of breath groups; n_syll: Average number of syllables; dur_g: average duration (± one standard error).

3.1. Linguistic structure of the breath group¶

The average characteristics of the breath groups and their repartition according to the first clause and to the number of clauses are displayed in Table 1. Speakers produced from 13 to 99 breath groups (47.4 (mean) ±4.5 (sterr), Figure 2). Half of the breath groups (53%) started with a matrix clause (m), a quarter (24%) with and embedded clause and the last quarter (23%) with an uncompleted clause (u). On average, the breath group included ~15.9 syllables (range: 1 to 50), and lasted ~3.5 s (range: .17 to 12.1). The number of syllables and the duration of the groups significantly increased with the number of clauses (~+7.5 syllables and +1.5 s per supplementary clause), but were similar for groups starting with a matrix as compared to an embedded clause. Groups starting with an uncompleted clause were ~6 syllables and 1.1 s shorter than the other groups.

Figure 2. Number of breath groups for each speaker with repartition of groups in: no hesitation (none), at least one hesitation at onset or elsewhere

The percentage of breath groups with vocalized hesitations ranged form 0 to more than 50% according to the subject (average 40%, see Figure 2). Among the breath groups with at least one hesitation (n=482), 40% started with a hesitation. Note that the groups with at least one hesitation not at the onset of the group were longer than the groups starting with a hesitation (~+3syllables and ~+749 ms) and than the groups without hesitation (~+3syllables and ~+1246 ms). The effect of hesitation type (t_hesi) on the number of syllables and the duration of the group were significant but didn’t interact with the effect of the first clause.

Figure 3. Correlations between n_syll in the breath group with: dur_I and amp_I and amp_IE, all values (top), average (bottom). Correlations values for amp_IE are indicated for log(amp_IE), see text for details.

3.2. Breathing kinematics¶

On average, the duration of inhalation was 676 ms (±8.5) and the amplitude was 17.6 %MD (± 0.2). The amplitude and the duration of inhalation depended both on the length of the breath group and on the type of the first clause. These values were also positively correlated with the number of syllables (r = ~.20 for all values and r = ~.60 for average correlations, see Figure 3, first two columns). LMM showed a significant effect of n_syll on both amp_I and dur_I.

Figure 4. Average and standard errors of dur_I, amp_I and amp_IE according to n_clauses and f_clause (white panels) and to the type of hesitation in the breath group (gray panel)

The duration of inhalation (Figure 4.A) significantly increased from 1 to 2 (+26 ms) and 2 to 3 (+36 ms) clauses. Dur_I was also longer for groups starting with a matrix clause as compared to other types of clauses (+197 ms). Inhalation (Figure 4.B) was significantly deeper when the first clause of the upcoming group was a matrix clause (+3.5 %MD) than any other clauses. Yet, amp_I did not significantly depend on the number of clauses. The analysis of the inhalation displacement relative to the exhalation displacement (amp_IE, Figure 3 and 4) shows: (1) that amp_IE was close to 1 for groups with 2 clauses and groups with 15-18 syllables; (2) a significant linear correlation between the logarithm of amp_IE with the number of syllables (-.48 for all values, -.83 for average, significant effect of n_syll); (3) an effect of the number of clauses (1 > 2 > 3); (4) no significant effect of the type of the first clause. Hence, on average, the inhalation displacement was similar to the exhalation displacement for groups with 2 clauses or 15-18 syllables, larger for shorter groups and smaller for longer groups. Inhalations were deeper (+2.54 %MD) and longer (+41 ms) for the breath groups with at least one hesitation as compare with no hesitation (Figure 4). The effect of t_hesi on amp_IE was not significant when the number of syllables was taken into account.

4. Discussion¶

The present study investigated the linguistic structure of the breath group in German spontaneous speech and evaluated if this structure is reflected in breathing kinematics. The important findings are:

• 1. Inhalations occur at syntactic boundaries (before a matrix or an embedded clause) or before a disfluency (uncompleted clause, repetition, hesitation, repair);
• 2. Inhalation depth and duration reflect: (2.1) the length of the breath group (number of syllables); (2.2) the type of the first clause, with deeper and longer inhalation for groups starting with a matrix clause as compared to the other groups; (2.3) vocalized hesitations, with deeper and longer inhalations for groups that include at least one vocalized hesitation as compared to none;
• 3. Syntactic complexity (number of clauses) is reflected only in the duration but not in the amplitude of inhalation;
• 4. On average the amplitude of exhalation is similar to the amplitude of inhalation for groups with 2 clauses or 15-18 syllables.

The observation that most of the inhalation pauses respect the syntactic organization of speech is consistent with previous work on English spontaneous speech [7,15]. The average duration (3.5 s), the number of syllables in the breath group (16 syllables) and the duration of inhalation (~.7 s) are also similar to values reported in the literature on English language ([7,8,15]).

As described in the introduction, previous studies found deeper and longer inhalations for longer utterances. Our dataset also show these relations. However, we also found that inhalations were deeper and longer for the breath groups starting with a matrix clause and for the groups including hesitations as compared to the other groups. To our knowledge, the relationship between the type of the first clause and hesitation to inhalation parameters have not been investigated so far for spontaneous speech. This relation is important with respect to the understanding of speech planning. It suggests that speaker inhale more air: (1) when they are starting a matrix clause that may come with other related clauses; (2) when they produce hesitations and do not know exactly what they are going to say. In this case, they can use vocalized hesitations as fillers during the exhalation phase, which could help to preserve ventilation and speech at the same time [21]. The fact that the breath groups with a hesitation at the onset were shorter than groups with a later hesitation shows that when hesitation came at the onset of the group, speaker probably inhaled again soon after it.

We also found that groups with an average number of syllables (15-18) show similar exhalation and inhalation amplitudes. These breath groups correspond to 2 clauses and could be a “favored” association between linguistic structure and breathing. This hypothesis should be tested by considering inter-speaker variability and speaker-specific lung volume capacities.

The speech task used in the present study required speakers to summarize the story they have just heard. This task is cognitively demanding and could have influenced the production of hesitations and the breathing profiles. This is in line with inter-speakers variability we found with respect to the number of breath groups and hesitations produced in the current task. To our knowledge only [8] have investigated the possible effect of cognitive load on breathing kinematics during spontaneous speech. We think it is important to distinguish between speaker-specific behaviors according to the task (e.g. variation in disfluency, hesitations).

5. Limits and perspectives¶

This study is a first analysis of a larger corpus of breathing kinematics in German spontaneous speech that now includes more than 50 speakers. Our global aim is to understand the interplay of speech planning and breathing in unconstrained speech. From the current study some first issues appear: (1) it is difficult to distinguish between the effect of the number of syllables and the effect of the number of clauses. Note that the quartile of the average number of syllables (10-15-21) were close to the average number of syllables in 1, 2, and 3 clauses, respectively (10-17-25 syllables); (2) Uncompleted clauses should be analyzed in more detail by splitting between hesitations, repairs and repetitions, that could have specific effect on breathing; (3) the amplitude of inhalation anticipates the upcoming breath group, but may also rely on what happened before [9]. This may be especially true for groups starting with an embedded clause. The next step is also to characterize the breath group in spontaneous speech not only as an individual unit but as a temporal sequence that depends on the preceding and following speech.

Speaker-specific behavior and context effects should also be considered. Previous studies on read and spontaneous speech, found that the properties of the breath group and their relations to inhalation parameters are speaker-specific [10,13], varied with age [11], cognitive load [8], speech rate [3] and loudness [16,19]. A large variability has also been observed for a same subject across repetitions and according to her emotional state [6-7, 10]. The sensitivity of speakers’ breathing regarding these multiple influences is important to understand the interplay between linguistics and respiration and may provide a fundamental tool for pathological diagnostics and speech therapy. Furthermore, implementing breathing in speech synthesis may improve the naturalness of speech synthesizers.

6. Acknowledgements¶

This work was funded by a grant from the BMBF (01UG0711) and the French-German University to the PILIOS project. The authors want to thanks Jörg Dreyer, Anna Sopronova and Uwe Reichel for their help with data collection and labeling.

7. References¶

• [1] Lieberman, P., Intonation, Perception and Language. (1967), Cambridge MA: MIT Press.
• [2] Henderson, A., Goldman-Eisler, F., & Skarbek, A. (1965). Temporal patterns of cognitive activity and breath control in speech. Lang Speech, 8, 236–242.
• [3] Grosjean, F. & Collins, M. (1979). Breathing, pausing and reading. Phonetica, 36(2), 98–114.
• [4] Conrad, B. & Schonle, P. (1979). Speech and respiration. Arch Psychiatr Nervenkr, 226, 251–268.
• [5] Conrad, B., Thalacker, S., & Schonle, P. (1983). Speech respiration as an indicator of integrative contextual processing. Folia Phoniatr (Basel), 35, 220–225.
• [6] Winkworth, A. L., Davis, P. J., Ellis, E., & Adams, R. D. (1994). Variability and consistency in speech breathing during reading: lung volumes, speech intensity, and linguistic factors. J Speech Hear Res, 37, 535–556.
• [7] Winkworth, A. L., Davis, P. J., Adams, R. D., & Ellis, E. (1995). Breathing patterns during spontaneous speech. J Speech Hear Res, 38, 124–144.
• [8] Mitchell, H. L., Hoit, J. D., & Watson, P. J. (1996). Cognitivelinguistic demands and speech breathing. J Speech Hear Res, 39, 93–104.
• [9] Bailly, G. and Gouvernayre, C. (2001). Pauses and respiratory markers of the structure of book reading. in Interspeech. 2012. Portland, OR.
• [10] Teston, B. and Autesserre, D. (1987). L’ aérodynamique du souffle phonatoire utilisé dans la lecture d’un texte en français. in International Congress of Phonetic Sciences (ICPhS). Estonia, University of Tallin. p. 33-36.
• [11] Sperry, E. E. & Klich, R. J. (1992). Speech breathing in senescent and younger women during oral reading. J Speech Hear Res, 35, 1246–1255.
• [12] Whalen, D. H. & Kinsella-Shaw, J. M. (1997). Exploring the relationship of inspiration duration to utterance duration. Phonetica, 54, 138–152.
• [13] Fuchs, S., Petrone, C. Krivokapic, J. & Hoole, P. (2013). Acoustic and respiratory evidence for utterance planning in German. Journal of Phonetics 41. 29-47.
• [14] McFarland, D. H. & Smith, A. (1992). Effects of vocal task and respiratory phase on prephonatory chest wall movements. J Speech Hear Res, 35, 971–982.
• [15] Wang, Y. T., Green, J. R., Nip, I. S., Kent, R. D., & Kent, J. F. (2010). Breath group analysis for reading and spontaneous speech in healthy adults. Folia Phoniatr Logop, 62, 297–302.
• [16] Huber, J. E. (2008). Effects of utterance length and vocal loudness on speech breathing in older adults}. Respir Physiol Neurobiol, 164, 323–330.
• [17] Wang, Y.T., Nip, I.S.B., Green, J.R., Kent, R.D., Kent, J.F., Ullman, C. Accuracy of perceptual and acoustic methods for the detection of inspiratory loci in spontaneous speech. Behavior
• Research Methods, 2012. 44(4): p. 1121-1128 [18] McFarland, D. H. (2001). Respiratory markers of conversational interaction}. J. Speech Lang. Hear. Res., 44, 128
• [19] Huber, J. E. (2007). Effect of cues to increase sound pressure level on respiratory kinematic patterns during connected speech}. J. Speech Lang. Hear. Res., 50, 621–634.
• [20] Ferreira, F. and Bailey K. G.D. (2004). Disfluencies and human language comprehension. Trends in Cognitive Sciences, 8(5), 231–237.
• [21] Schonle, P. W. & Conrad, B. (1985). Hesitation vowels: a motor speech respiration hypothesis. Neurosci. Lett., 55, 293–296.
• [22] Konno, K. & Mead, J. (1967). Measurement of the separate volume changes of rib cage and abdomen during breathing. Journal of Applied Physiology 22(3), 407–422.
• [23] Banzett, R. B., Mahan, S. T., Garner, D. M., Brughera, A. & Loring, S. H. (1995). A simple and reliable method to calibrate respiratory magnetometers and Respitrace. Journal of Applied Physiology, 79(6), 2169-2176.
• [24] Boersma, P. and D. Weenink, Praat, a System for doing Phonetics by Computer, version 3.4, in Institute of Phonetic Sciences of the University of Amsterdam, Report 132. 182 pages. 1996.
• [25] Reichel, U.D. (2012). PermA and Balloon: Tools for string alignment and text processing. Proceedings of Interspeech, Portland, paper 346.

Abstract¶

Learning phonetic categories is one of the first steps to learning a language, yet is hard to do using only distributional phonetic information. Semantics could potentially be useful, since words with different meanings have distinct phonetics, but it is unclear how many word meanings are known to infants learning phonetic categories. We show that attending to a weaker source of semantics, in the form of a distribution over topics in the current context, can lead to improvements in phonetic category learning. In our model, an extension of a previous model of joint word-form and phonetic category inference, the probability of word-forms is topic-dependent, enabling the model to find significantly better phonetic vowel categories and word-forms than a model with no semantic knowledge.

1 Introduction¶

Infants begin learning the phonetic categories of their native language in their first year (Kuhl et al., 1992; Polka and Werker, 1994; Werker and Tees, 1984). In theory, semantic information could offer a valuable cue for phoneme induction [1] by helping infants distinguish between minimal pairs, as linguists do (Trubetzkoy, 1939). However, due to a widespread assumption that infants do not know the meanings of many words at the age when they are learning phonetic categories (see Swingley, 2009 for a review), most recent models of early phonetic category acquisition have explored the phonetic learning problem in the absence of semantic information (de Boer and Kuhl, 2003; Dillon et al., 2013; Feldman et al., 2013a; McMurray et al., 2009; Vallabha et al., 2007).

Models without any semantic information are likely to underestimate infants’ ability to learn phonetic categories. Infants learn language in the wild, and quickly attune to the fact that words have (possibly unknown) meanings. The extent of infants’ semantic knowledge is not yet known, but existing evidence shows that six-month-olds can associate some words with their referents (Bergelson and Swingley, 2012; Tincoff and Jusczyk, 1999, 2012), leverage non-acoustic contexts such as objects or articulations to distinguish similar sounds (Teinonen et al., 2008; Yeung and Werker, 2009), and map meaning (in the form of objects or images) to new word-forms in some laboratory settings (Friedrich and Friederici, 2011; Gogate and Bahrick, 2001; Shukla et al., 2011). These findings indicate that young infants are sensitive to co-occurrences between linguistic stimuli and at least some aspects of the world.

In this paper we explore the potential contribution of semantic information to phonetic learning by formalizing a model in which learners attend to the word-level context in which phones appear (as in the lexical-phonetic learning model of Feldman et al., 2013a) and also to the situations in which word-forms are used. The modeled situations consist of combinations of categories of salient activities or objects, similar to the activity contexts explored by Roy et al. (2012), e.g., ‘getting dressed’ or ‘eating breakfast’. We assume that child learners are able to infer a representation of the situational context from their non-linguistic environment. However, in our simulations we approximate the environmental information by running a topic model (Blei et al., 2003) over a corpus of childdirected speech to infer a topic distribution for each situation. These topic distributions are then used as input to our model to represent situational contexts.

The situational information in our model is similar to that assumed by theories of cross-situational word learning (Frank et al., 2009; Smith and Yu, 2008; Yu and Smith, 2007), but our model does not require learners to map individual words to their referents. Even in the absence of word-meaning mappings, situational information is potentially useful because similar-sounding words uttered in similar situations are more likely to be tokens of the same lexeme (containing the same phones) than similarsounding words uttered in different situations.

In simulations of vowel learning, inspired by Vallabha et al. (2007) and Feldman et al. (2013a), we show a clear improvement over previous models in both phonetic and lexical (word-form) categorization when situational context is used as an additional source of information. This improvement is especially noticeable when the word-level context is providing less information, arguably the more realistic setting. These results demonstrate that relying on situational co-occurrence can improve phonetic learning, even if learners do not yet know the meanings of individual words.

注釈

2 Background and overview of models¶

Infants attend to distributional characteristics of their input (maye et al., 2002, 2008), leading to the hypothesis that phonetic categories could be acquired on the basis of bottom-up distributional learning alone (de Boer and Kuhl, 2003; Vallabha et al., 2007; McMurray et al., 2009). However, this would require sound categories to be well separated, which often is not the case—for example, see :num:`Figure #fig1`, which shows the English vowel space that is the focus of this paper.

英語母音空間(Hillenbrand ら(1995) の6.2節より引用).第一,第二フォルマントを図示する.

Recent work has investigated whether infants could overcome such distributional ambiguity by incorporating top-down information, in particular, the fact that phones appear within words. At six months, infants begin to recognize word-forms such as their name and other frequently occurring words (Mandel et al., 1995; Jusczyk and Hohne, 1997), without necessarily linking a meaning to these forms. This ‘protolexicon’ can help differentiate phonetic categories by adding word contexts in which certain sound categories appear (Swingley, 2009; Feldman et al., 2013b). To explore this idea further, Feldman et al. (2013a) implemented the Lexical-Distributional (LD) model, which jointly learns a set of phonetic vowel categories and a set of word-forms containing those categories. Simulations showed that the use of lexical context greatly improved phonetic learning.

Our own Topic-Lexical-Distributional (TLD) model extends the LD model to include an additional type of context: the situations in which words appear. To motivate this extension and clarify the differences between the models, we now provide a high-level overview of both models; details are given in Sections 3 and 4.

2.1 Overview of LD model¶

Both the LD and TLD models are computationallevel models of phonetic (specifically, vowel) categorization where phones (vowels) are presented to the model in the context of words. [2] The task is to infer a set of phonetic categories and a set of lexical items on the basis of the data observed for each word token \(x_i\) . In the original LD model, the observations for token \(x_i\) are its frame \(f_i\) , which consists of a list of consonants and slots for vowels, and the list of vowel tokens \(w_i\). (The TLD model includes additional observations, described below.) A single vowel token, \(w_{ij}\) , is a two dimensional vector representing the first two formants (peaks in the frequency spectrum, ordered from lowest to highest). For example, a token of the word kitty would have the frame \(f_i = \text{k\_t\_}\) , containing two consonant phones, /k/ and /t/, with two vowel phone slots in between, and two vowel formant vectors, \(w_{i0} = [464, 2294] \text{ and } w_{i1} = [412, 2760]\). [3]

Given the data, the model must assign each vowel token to a vowel category, \(w_{ij} = c\). Both the LD and the TLD models do this using intermediate lexemes, \(\ell\) , which contain vowel category assignments, \(\nu_{\ell j} = c\), as well as a frame \(f_{\ell}\) . If a word token is assigned to a lexeme, \(x_i = \ell\), the vowels within the word are assigned to that lexeme’s vowel categories, \(w_{ij} = \nu_{\ell j} = c\) . [4] The word and lexeme frames must match, \(f_i = f_{\ell}\) .

Lexical information helps with phonetic categorization because it can disambiguate highly overlapping categories, such as the \(ae\) and \(eh\) categories in :num:`Figure #fig1`. A purely distributional learner who observes a cluster of data points in the \(ae-eh\) region is likely to assume all these points belong to a single category because the distributions of the categories are so similar. However, a learner who attends to lexical context will notice a difference: contexts that only occur with \(ae\) will be observed in one part of the \(ae-eh\) region, while contexts that only occur with \(eh\) will be observed in a different (though partially overlapping) space. The learner then has evidence of two different categories occurring in different sets of lexemes.

Simulations with the LD model show that using lexical information to constrain phonetic learning can greatly improve categorization accuracy (Feldman et al., 2013a), but it can also introduce errors. When two word tokens contain the same consonant frame but different vowels (i.e., minimal pairs), the model is more likely to categorize those two vowels together. Thus, the model has trouble distinguishing minimal pairs. Although young children also have trouble with minimal pairs (Stager and Werker, 1997; Thiessen, 2007), the LD model may overestimate the degree of the problem. We hypothesize that if a learner is able to associate words with the contexts of their use (as children likely are), this could provide a weak source of information for disambiguating minimal pairs even without knowing their exact meanings. That is, if the learner hears \(k V_1 t\) and \(k V_2 t\) in different situational contexts, they are likely to be different lexical items (and \(V_1\) and \(V_2\) different phones), despite the lexical similarity between them.

脚注

2.2 Overview of TLD model¶

To demonstrate the benefit of situational information, we develop the Topic-Lexical-Distributional (TLD) model, which extends the LD model by assuming that words appear in situations analogous to documents in a topic model. Each situation \(h\) is associated with a mixture of topics \(theta_h\) , which is assumed to be observed. Thus, for the \(i\) th token in situation \(h\), denoted \(x_{hi}\) , the observed data will be its frame \(f_{hi}\) , vowels \(w_{hi}\) , and topic vector \(\theta_h\) .

From an acquisition perspective, the observed topic distribution represents the child’s knowledge of the context of the interaction: she can distinguish bathtime from dinnertime, and is able to recognize that some topics appear in certain contexts (e.g. animals on walks, vegetables at dinnertime) and not in others (few vegetables appear at bathtime). We assume that the child would learn these topics from observing the world around her and the co-occurrences of entities and activities in the world. Within any given situation, there might be a mixture of different (actual or possible) topics that are salient to the child. We assume further that as the child learns the language, she will begin to associate specific words with each topic as well.

Thus, in the TLD model, the words used in a situation are topic-dependent, implying meaning, but without pinpointing specific referents. Although the model observes the distribution of topics in each situation (corresponding to the child observing her non-linguistic environment), it must learn to associate each (phonetically and lexically ambiguous) word token with a particular topic from that distribution. The occurrence of similar-sounding words in different situations with mostly non-overlapping topics will provide evidence that those words belong to different topics and that they are therefore different lexemes. Conversely, potential minimal pairs that occur in situations with similar topic distributions are more likely to belong to the same topic and thus the same lexeme.

Although we assume that children infer topic distributions from the non-linguistic environment, we will use transcripts from CHILDES to create the word/phone learning input for our model. These transcripts are not annotated with environmental context, but Roy et al. (2012) found that topics learned from similar transcript data using a topic model were strongly correlated with immediate activities and contexts. We therefore obtain the topic distributions used as input to the TLD model by training an LDA topic model (Blei et al., 2003) on a superset of the child-directed transcript data we use for lexical-phonetic learning, dividing the transcripts into small sections (the ‘documents’ in LDA) that serve as our distinct situations \(h\) . As noted above, the learned document-topic distributions \(\theta\) are treated as observed variables in the TLD model to represent the situational context. The topic-word distributions learned by LDA are discarded, since these are based on the (correct and unambiguous) words in the transcript, whereas the TLD model is presented with phonetically ambiguous versions of these word tokens and must learn to disambiguate them and associate them with topics.

訳者注

3 Lexical-Distributional Model¶

In this section we describe more formally the generative process for the LD model (Feldman et al., 2013a), a joint Bayesian model over phonetic categories and a lexicon, before describing the TLD extension in the following section.

The set of phonetic categories and the lexicon are both modeled using non-parametric Dirichlet Process priors, which return a potentially infinite number of categories or lexemes. A DP is parametrized as \(DP (\alpha, H)\), where \(\alpha\) is a real-valued hyperparameter and \(H\) is a base distribution. \(H\) may be continuous, as when it generates phonetic categories in formant space, or discrete, as when it generates lexemes as a list of phonetic categories.

A draw from a \(DP, G ∼ DP (\alpha, H)\) , returns a distribution over a set of draws from \(H\), i.e., a discrete distribution over a set of categories or lexemes generated by \(H\). In the mixture model setting, the category assignments are then generated from \(G\), with the datapoints themselves generated by the corresponding components from \(H\). If \(H\) is infinite, the support of the \(DP\) is likewise infinite. During inference, we marginalize over \(G\).

4 Topic-Lexical-Distributional Model¶

The TLD model retains the IGMM vowel phone component, but extends the lexicon of the LD model by adding topic-specific lexicons, which capture the notion that lexeme probabilities are topicdependent. Specifically, the TLD model replaces the Dirichlet Process lexicon with a Hierarchical Dirichlet Process (HDP; Teh (2006)). In the HDP lexicon, a top-level global lexicon is generated as in the LD model. Topic-specific lexicons are then drawn from the global lexicon, containing a subset of the global lexicon (but since the size of the global lexicon is unbounded, so are the topic-specific lexicons). These topic-specific lexicons are used to generate the tokens in a similar manner to the LD model. There are a fixed number of lower level topic-lexicons; these are matched to the number of topics in the LDA model used to infer the topic distributions (see Section 6.4).

More formally, the global lexicon is generated as a top-level \(DP: G_L ∼ DP (\alpha_{\ell} , H_L )\) (see Section 3.2; remember \(H_L\) includes draws from the IGMM over vowel categories). \(G_L\) is in turn used as the base distribution in the topic-level DPs, \(G_k ∼ DP (\alpha_k , G_L )\). In the Chinese Restaurant Franchise metaphor often used to describe HDPs, \(G_L\) is a global menu of dishes (lexemes). The topicspecific lexicons are restaurants, each with its own distribution over dishes; this distribution is defined by seating customers (word tokens) at tables, each of which serves a single dish from the menu: all tokens \(x\) at the same table \(t\) are assigned to the same lexeme \(t\) . Inference (Section 5) is defined in terms of tables rather than lexemes; if multiple tables draw the same dish from \(G_L\) , tokens at these tables share a lexeme.

In the TLD model, tokens appear within situations, each of which has a distribution over topics \(\theta_h\) . Each token \(x_{hi}\) has a co-indexed topic assignment variable, \(z_{hi}\) , drawn from \(\theta_h\) , designating the topic-lexicon from which the table for \(x_{hi}\) is to be drawn. The formant values for \(w_{hij}\) are drawn in the same way as in the LD model, given the lexeme assignment at \(x_{hi}\) . This results in the following model, shown in :num:`Figure #fig2`:

左から右へ TDL モデル, IGMMコンポーネント, LD 語彙コンポーネント, 話題に限定的な語彙, 最後に文章 \(h\) に現れるトークン \(x_{hi}\) と 観察された母音フォルマント \(w_{hij}\) と フレーム \(f_{hi}\)

The lexeme assignment \(x_{hi}\) and the topic assignment \(z_{hi}\) are inferred, the latter using the observed documenttopic distribution \(theta_h\) . Note that \(f_i\) is deterministic given the lexeme assignment. Squared nodes depict hyperparameters. \(\lambda\) is the set of hyperparameters used by \(H_L\) when generating lexical items (see Section 3.2).

5 Inference: Gibbs Sampling¶

We use Gibbs sampling to infer three sets of variables in the TLD model: assignments to vowel categories in the lexemes, assignments of tokens to topics, and assignments of tokens to tables (from which the assignment to lexemes can be read off).

6 Experiments¶

6.6 Results¶

We compare all three models—TLD, LD, and IGMM—on the vowel categorization task, and TLD and LD on the lexical categorization task (since IGMM does not infer a lexicon). The datasets correspond to two sets of conditions: firstly, either using vowel categories synthesized from all speakers or only adult female speakers, and secondly, varying the coarseness of the observed consonant categories. Each condition (model, vowel speakers, consonant set) is run five times, using 1500 iterations of Gibbs sampling with hyperparameter sampling. Overall, we find that TLD outperforms the other models in both tasks, across all conditions.

注釈

我々は 3つのモデル —TLD, LD, IGMM— すべてを母音分類タスク,で比較し, TLD モデル と LDモデル を語彙の分類タスクで比較した( IGMM は語彙を推定しないため). データセットは2組の条件に対応している. まず, すべてのまたは,女性のみの話者のどちらかから合成された母音カテゴリを使用する条件. つづいて, 観察された子音のカテゴリの粗さを変化させる条件である. 各条件 (モデル, 母音話者, 子音のセット) は五回ずつ実行され, ハイパラメータサンプリングをもつ Gibbd サンプリング の反復を 1500 回行った. 総括すると TLD モデルはすべての条件,両方のタスクで他のモデルよりも良い性能であることが分かった.

Vowel categorization results are shown in :num:`Figure #fig3`. IGMM performs substantially worse than both TLD and LD, with scores more than 30 points lower than the best results for these models, clearly showing the value of the protolexicon and repli- cating the results found by Feldman et al. (2013a) on this dataset. Furthermore, TLD consistently outperforms the LD model, finding better phonetic categories, both for vowels generated from the combined categories of all speakers (‘all’) and vowels generated from adult female speakers only (‘w’), although the latter are clearly much easier for both models to learn. Both models perform less well when the consonant frames provide less information, but the TLD model performance degrades less than the LD performance.

注釈

母音のカテゴリ化の結果を :num:`図 #fig3` に示す. IGMMは常に TLD, LD 双方より悪い成績であり, このデータセット上の Feldman et al. (2013a) で発見された結果の反復であり, はっきりと 前-語彙の値を示しているこれらのモデルの最もよい値より 30 ポイント以上スコアが低い. 更に, TLD モデルは常に LD モデルよりも性能がよく, 全てのスピーカーの組み合わせのカテゴリから生成された母音’all’, 成人女性のみの発話だけから生成された母音 ‘W’ 両方で, 後者の方が2つのモデルともに学習が容易であるが, より良い音素カテゴリを発見できた. 子音フレーム提供する情報がが少ない場合, 両モデルの機能は低下する. しかし, この低下はTLDモデルの方が, LD モデルよりも少ない.

母音の評価

• ‘all’ : すべての話者からの合成した母音セットを使用
• ‘w’ : 成人女性の母音から合成した母音セットを使用

バーは五回の実行を基準に95%信頼区間を示す. ここには示していないが, IGMMの結果は以下の通りである.

• IGMM-all : VM score of 53.9 (CI=0.5)
• IGMM-w : VM score of 65.0 (CI=0.2)

The gold-standard vowels are shown in gold in the background but are mostly overlapped by the inferred categories.

注釈

ゴールドスタンダードの母音は、バックグラウンドに金色で示されているが、ほとんどは推論されたカテゴリ毎に重なっている.

Both the TLD and the LD models find ‘supervowel’ categories, which cover multiple vowel categories and are used to merge minimal pairs into a single lexical item. :num:`Figure #fig4` shows example vowel categories inferred by the TLD model, including two supervowels. The TLD supervowels are used much less frequently than the supervowels found by the LD model, containing, on average, only twothirds as many tokens.

注釈

TLD , LD モデル 両方とも 複数の母音カテゴリにまたがり,一つの語彙的アイテムに含まれるミニマルペアを結合している “上位母音” カテゴリを発見している. :num:`図 #fig4` に TDL モデルによって推定された2つの上位母音を含む母音カテゴリの例を示す. TDL モデルでの上位母音は平均してトークンの \(\frac{2}{3}\) 程度の, LDモデルが発見した上位母音より少ない頻度が使用されている.

TLD モデルによって発見された母音:

上位母音を赤く示す. 正解の母音は背景に黄色で示すが,推定されたカテゴリによっておおよそが書き換えられている.

:num:`Figure #fig5` shows that TLD also outperforms LD on the lexeme/word categorization task. Again performance decreases as the consonant categories become coarser, but the additional semantic information in the TLD model compensates for the lack of consonant information. In the individual components of VM, TLD and LD have similar VC (‘recall’), but TLD has higher VH (‘precision’), demonstrating that the semantic information given by the topics can separate potentially ambiguous words, as hypothesized.

注釈

:num:`図 #fig5` に TLD モデルが 語彙素性/単語のカテゴリ化タスクで LD モデルよりも優れていることを示す. ここでも,子音のカテゴリは、粗くなればなるほど,パフォーマンスの低下が見られるが, TLD モデルにおける意味情報の追加は子音情報の不足を補償している. VMの個々の成分を確認するとTLDとLDは、類似したVC（’recall’）を持っているが, TLDは、より高いVH（’precision’）を有している. この結果は話題によって与えられた意味情報が、潜在的にあいまいな言葉を分離することができるという仮説を実証するものである.

語彙素性の評価

• ‘all’ : すべての話者から合成された母音を含むデータセットを使用
• ‘w’ : 成人女性の母音から合成された母音を含むデータセットを使用

Overall, the contextual semantic information added in the TLD model leads to both better phonetic categorization and to a better protolexicon, especially when the input is noisier, using degraded consonants. Since infants are not likely to have perfect knowledge of phonetic categories at this stage, semantic information is a potentially rich source of information that could be drawn upon to offset noise from other domains. The form of the semantic information added in the TLD model is itself quite weak, so the improvements shown here are in line with what infant learners could achieve.

注釈

結論として, TLD モデルで追加された文脈の意味情報は音素カテゴリ化と 源-語彙 の両方をより良くする. これは子音情報が劣化してノイズが多いときには特に顕著になる. 乳児は、この段階で音素カテゴリの完全な知識を持っていそうにないため, 意味情報は,潜在的に他のドメインからのノイズをうまく相殺することのできる豊かな情報源なのである. TLDモデルで追加された文脈の意味情報はそれ自体は非常に弱いが, ここに示した改善は幼児学習者が達成できるものと一致している.

7 Conclusion¶

Language acquisition is a complex task, in which many heterogeneous sources of information may be useful. In this paper, we investigated whether contextual semantic information could be of help when learning phonetic categories. We found that this contextual information can improve phonetic learning performance considerably, especially in situations where there is a high degree of phonetic ambiguity in the word-forms that learners hear. This suggests that previous models that have ignored semantic information may have underestimated the information that is available to infants. Our model illustrates one way in which language learners might harness the rich information that is present in the world without first needing to acquire a full inventory of word meanings.

The contextual semantic information that the TLD model tracks is similar to that potentially used in other linguistic learning tasks. Theories of cross-situational word learning (Smith and Yu, 2008; Yu and Smith, 2007) assume that sensitivity to situational co-occurrences between words and non-linguistic contexts is a precursor to learning the meanings of individual words. Under this view, contextual semantics is available to infants well before they have acquired large numbers of semantic minimal pairs. However, recent experimental evidence indicates that learners do not always retain detailed information about the referents that are present in a scene when they hear a word (Medina et al., 2011; Trueswell et al., 2013). This evidence poses a direct challenge to theories of cross-situational word learning. Our account does not necessarily require learners to track co-occurrences between words and individual objects, but instead focuses on more abstract information about salient events and topics in the environment; it will be important to investigate to what extent infants encode this information and use it in phonetic learning.

Regardless of the specific way in which infants encode semantic information, our method of adding this information by using LDA topics from transcript data was shown to be effective. This method is practical because it can approximate semantic information without relying on extensive manual annotation.

The LD model extended the phonetic categorization task by adding word contexts; the TLD model presented here goes even further, adding larger situational contexts. Both forms of top-down information help the low-level task of classifying acoustic signals into phonetic categories, furthering a holistic view of language learning with interaction across multiple levels.

Acknowledgments¶

This work was supported by EPSRC grant EP/H050442/1 and a James S. McDonnell Foundation Scholar Award to the final author.

References¶

• Frans Adriaans and Daniel Swingley.

  • Distributional learning of vowel categories is supported by prosody in infant-directed speech.
  • In Proceedings of the 34th Annual Conference of the Cognitive Science Society (CogSci), 2012.

• 5. Bergelson and D. Swingley.

  • At 6-9 months, human infants know the meanings of many common nouns.
  • Proceedings of the National Academy of Sciences, 109(9) : 3253-3258, Feb 2012.

• David M. Blei, Thomas L. Griffiths, Michael I. Jordan, and Joshua B. Tenenbaum.

  • Hierarchical topic models and the nested Chinese restaurant process.
  • In Advances in Neural Information Processing Systems 16, 2003.

• Michael R. Brent and Jeffrey M. Siskind.

  • The role of exposure to isolated words in early vocabulary development.
  • Cognition, 81(2):B33–B44, 2001.

• Christos Christodoulopoulos, Sharon Goldwater, and Mark Steedman.

  • Two decades of unsupervised POS induction: How far have we come?
  • In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 575–584, Cambridge, MA, October 2010.
  • Association for Computational Linguistics.

• Bart de Boer and Patricia K. Kuhl.

  • Investigating the role of infant-directed speech with a computer model.
  • Acoustics Research Letters Online, 4(4): 129, 2003.

• Brian Dillon, Ewan Dunbar, and William Idsardi.

  • A single-stage approach to learning phonological categories: Insights from Inuktitut.
  • Cognitive Science, 37(2):344–377, Mar 2013.

• Micha Elsner, Sharon Goldwater, Naomi Feldman, and Frank Wood.

  • A cognitive model of early lexical acquisition with phonetic variability.
  • In Proceedings of the 18th Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013.
  • word learning.
  • Psychological Science, 20(5): 578–585, 2009.

• Manuela Friedrich and Angela D. Friederici.

  • Word learning in 6-month-olds: Fast encoding—weak retention.
  • Journal of Cognitive Neuroscience, 23 (11):3228–3240, Nov 2011.

• Lakshmi J. Gogate and Lorraine E. Bahrick.

  • Intersensory redundancy and 7-month-old infants’ memory for arbitrary syllable-object relations.
  • Infancy, 2(2):219–231, Apr 2001.

• 10. Hillenbrand, L. A. Getty, M. J. Clark, and K. Wheeler.

  • Acoustic characteristics of American English vowels.
  • Journal of the Acoustical Society of America, 97(5 Pt 1):3099–3111, May 1995.

• 16. 23. Jusczyk and Elizabeth A. Hohne.

  • Infants’ memory for spoken words.
  • Science, 277(5334): 1984–1986, Sep 1997.

• Patricia K. Kuhl, Karen A. Williams, Francisco Lacerda, Kenneth N. Stevens, and Bjorn Lindblom.

  • Linguistic experience alters phonetic perception in infants by 6 months of age.
  • Science, 255(5044):606–608, 1992.

• Brian MacWhinney.

  • The CHILDES Project: Tools for Analyzing Talk.
  • Lawrence Erlbaum Associates, 2000.

• 4. 18. Mandel, P. W. Jusczyk, and D. B. Pisoni.

  • Infants’ recognition of the sound patterns of their own names.
  • Psychological Science, 6(5):314– 317, Sep 1995.

• Nivedita Mani and Kim Plunkett.

  • Twelve-montholds know their cups from their keps and tups.
  • Infancy, 15(5):445470, Sep 2010.

• Naomi H. Feldman, Thomas L. Griffiths, Sharon Goldwater, and James L. Morgan.

  • A role for the developing lexicon in phonetic category acquisition.
  • Psychological Review, 2013a.

• Jessica Maye, Daniel J. Weiss, and Richard N. Aslin.

  • Statistical phonetic learning in infants: facilitation and feature generalization.
  • Developmental Science, 11(1):122–134, Jan 2008.

• Naomi H. Feldman, Emily B. Myers, Katherine S. White, Thomas L. Griffiths, and James L. Morgan.

  • Word-level information influences phonetic learning in adults and infants.
  • Cognition, 127(3): 427–438, 2013b.

• Jessica Maye, Janet F Werker, and LouAnn Gerken.

  • Infant sensitivity to distributional information can affect phonetic discrimination.
  • Cognition, 82(3):B101–B111, Jan 2002.

• Abdellah Fourtassi and Emmanuel Dupoux. A rudimentary lexicon and semantics help bootstrap phoneme acquisition. Submitted.

• Michael C. Frank, Noah D. Goodman, and Joshua B. Tenenbaum. Using speakers’ referential intentions to model early cross-situational

• Andrew McCallum. MALLET: A machine learning for language toolkit, 2002.

• Bob McMurray, Richard N. Aslin, and Joseph C. Toscano. Statistical learning of phonetic categories: insights from a computational approach. Developmental Science, 12(3):369–378, May 2009.

• Tamara Nicol Medina, Jesse Snedeker, John C.

• Trueswell, and Lila R. Gleitman. How words

• can and cannot be learned by observation. Proceedings of the National Academy of Sciences,

• 108(22):9014–9019, 2011.

• Radford Neal. Markov chain sampling methods

• for Dirichlet process mixture models. Journal

• of Computational and Graphical Statistics, 9:

• 249–265, 2000.

• Linda Polka and Janet F. Werker. Developmental changes in perception of nonnative vowel

• contrasts. Journal of Experimental Psychology: Human Perception and Performance, 20(2):421– 435, 1994. ceedings of the 44th Annual Meeting of the Association for Computational Linguistics (ACL), pages 985 – 992, Sydney, 2006.

• Tuomas Teinonen, Richard N. Aslin, Paavo Alku, and Gergely Csibra. Visual speech contributes to phonetic learning in 6-month-old infants. Cognition, 108:850–855, 2008.

• Erik D. Thiessen. The effect of distributional information on children’s use of phonemic contrasts. Journal of Memory and Language, 56(1):16–34, Jan 2007.

• 18. Tincoff and P. W. Jusczyk. Some beginnings of word comprehension in 6-month-olds. Psychological Science, 10(2):172–175, Mar 1999.

• Carl Rasmussen. The infinite Gaussian mixture model. In Advances in Neural Information Processing Systems 13, 2000.

• Ruth Tincoff and Peter W. Jusczyk. Six-montholds comprehend words that refer to parts of the body. Infancy, 17(4):432444, Jul 2012.

• Andrew Rosenberg and Julia Hirschberg. Vmeasure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 12th Conference on Empirical Methods in Natural Language Processing (EMNLP), 2007.

• 14. 19. Trubetzkoy. Grundz¨ ge der Phonologie. Vanu denhoeck und Ruprecht, G¨ ttingen, 1939. o

• Brandon C. Roy, Michael C. Frank, and Deb Roy. Relating activity contexts to early word learning in dense longitudinal data. In Proceedings of the 34th Annual Conference of the Cognitive Science Society (CogSci), 2012.

• Rushen Shi and Janet F. Werker. The basis of preference for lexical words in 6-month-old infants. Developmental Science, 6(5):484–488, 2003.

• 13. Shukla, K. S. White, and R. N. Aslin. Prosody guides the rapid mapping of auditory word forms onto visual objects in 6-mo-old infants. Proceedings of the National Academy of Sciences, 108 (15):6038–6043, Apr 2011.

• Linda B. Smith and Chen Yu. Infants rapidly learn word-referent mappings via cross-situational statistics. Cognition, 106(3):1558–1568, 2008.

• Christine L. Stager and Janet F. Werker. Infants listen for more phonetic detail in speech perception than in word-learning tasks. Nature, 388: 381–382, 1997.

• 4. Swingley. Contributions of infant word learning to language development. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1536):3617–3632, Nov 2009.

• Yee Whye Teh. A hierarchical Bayesian language model based on Pitman-Yor processes. In Pro-

• John C. Trueswell, Tamara Nicol Medina, Alon Hafri, and Lila R. Gleitman. Propose but verify: Fast mapping meets cross-situational word learning. Cognitive Psychology, 66:126–156, 2013.

• 7. 11. Vallabha, J. L. McClelland, F. Pons, J. F. Werker, and S. Amano. Unsupervised learning of vowel categories from infant-directed speech. Proceedings of the National Academy of Sciences, 104(33):13273–13278, Aug 2007.

• Janet F. Werker and Richard C. Tees. Crosslanguage speech perception: Evidence for perceptual reorganization during the first year of life. Infant Behavior and Development, 7:49–63, 1984.

• 8. Henny Yeung and Janet F. Werker. Learning words’ sounds before learning how words sound: 9-month-olds use distinct objects as cues to categorize speech information. Cognition, 113(2): 234–243, Nov 2009.

• Chen Yu and Linda B. Smith.

  • Rapid word learning under uncertainty via cross-situational statistics.
  • Psychological Science, 18(5):414–420, 2007.

References and Notes¶

1. 14. Chomsky, M. Halle, The Sound Pattern of English (Harper and Row, New York, 1968).
2. 10. 18. Binder et al., Cereb. Cortex 10, 512–528 (2000).
3. 4. Boatman, C. Hall, M. H. Goldstein, R. Lesser, B. Gordon, Cortex 33, 83–98 (1997).
4. 5. 6. Chang et al., Nat. Neurosci. 13, 1428–1432 (2010).
5. 5. Formisano, F. De Martino, M. Bonte, R. Goebel, Science 322, 970–973 (2008).
6. 10. Obleser, A. M. Leaver, J. Vanmeter, J. P. Rauschecker, Front. Psychol. 1, 232 (2010).
7. 13. Steinschneider et al., Cereb. Cortex 21, 2332–2347 (2011).
8. Materials and methods are available as supplementary materials on Science Online.
9. 14. 5. Crone, D. Boatman, B. Gordon, L. Hao, Clin. Neurophysiol. 112, 565–582 (2001).
10. 5. Edwards et al., J. Neurophysiol. 102, 377–386 (2009).
11. 13. Steinschneider, Y. I. Fishman, J. C. Arezzo, Cereb. Cortex 18, 610–625 (2008).
12. 19. Ray, J. H. R. Maunsell, PLOS Biol. 9, e1000610 (2011).
13. 10. 19. Garofolo, TIMIT: Acoustic-Phonetic Continuous Speech Corpus (Linguistic Data Consortium, Philadelphia, 1993).
14. 6. 5. Theunissen et al., Network 12, 289–316 (2001).
15. 13. Halle, K. Stevens, in Music, Language, Speech, and Brain, J. Sundberg, L. Nord, R. Carlson, Eds. (Wenner-Gren International Symposium Series vol. 59, Macmillan, Basingstoke, UK, 1991).
16. 16. Ladefoged, K. Johnson, A Course in Phonetics (Cengage Learning, Stamford, CT, 2010).
17. 11. 14. Stevens, J. Acoust. Soc. Am. 111, 1872–1891 (2002).
18. 7. Clements, Phonol. Yearb. 2, 225–252 (1985).
19. 3. 1. Fowler, J. Phonetics 14, 3–28 (1986).
20. 7. 5. Peterson, H. L. Barney, J. Acoust. Soc. Am. 24, 175 (1952).
21. 10. 4. Miller, J. Acoust. Soc. Am. 85, 2114–2134 (1989).
22. 1. 11. Syrdal, H. S. Gopal, J. Acoust. Soc. Am. 79, 1086–1100 (1986).
23. 8. 13. Sussman, Brain Lang. 28, 12–23 (1986).
24. 1. Nelken, Curr. Opin. Neurobiol. 18, 413–417 (2008).
25. 14. Mesgarani, S. V. David, J. B. Fritz, S. A. Shamma, J. Acoust. Soc. Am. 123, 899–909 (2008).
26. 3. 20. Engineer et al., Nat. Neurosci. 11, 603–608 (2008).
27. 12. Lisker, A. S. Abramson, Lang. Speech 10, 1–28 (1967).
28. 13. Steinschneider et al., Cereb. Cortex 15, 170–186 (2005).
29. 7. Chechik, I. Nelken, Proc. Natl. Acad. Sci. U.S.A. 109, 18968–18973 (2012).
30. 1. 13. Chan et al., Cereb. Cortex, published online 16 May 2013 (10.1093/cercor/bht127).
31. 15. Creutzfeldt, G. Ojemann, E. Lettich, Exp. Brain Res. 77, 451–475 (1989).
32. 7. 1. Miller, P. E. Nicely, J. Acoust. Soc. Am. 27, 338 (1955).
33. 1. 13. Liberman, Speech: A Special Code (MIT Press, Cambridge, MA, 1996).

ABSTRACT¶

我々は音響的感情認識の特徴量選択におけるデータマイニング実験を示す。 ピッチ、エネルギー, MFCC 時間系列 に由来する 1000 個以上の特徴量から始め、 相関の高い特徴量を排除することで,このセットの中からデータに対し関連の高いものを選択した。 特徴量は演技音声、あるいは実際の感情を含む音声別に解析され、有意差が確認された。 全ての特徴量は自動的に計算され、自動で解析したものと手動で解析したものを比較した。 自動化の程度が高いものでも、認識精度の観点からでは、特に不利になることは無かった。

• This work was partially funded by a grant from the DFG in the graduate program 256 and by the EU Network of Excellence Humaine.

1.INTRODUCTION¶

音声から感情を認識するための多くの特徴量が発見されている。 しかし、一般に認められる一定の特徴量セットは未だ決まっていない。 我われはデータマイニングを行い、データのピッチ、エネルギー、MFCC 時系列における異なった視点を提供する音響特徴量の大規模なセットを計算した。 続いて、与えられたデータセットから最もよいサブセットを自動的に選択した。 このアプローチは音声感情認識の領域では一般的なものである [1] [2] [3] 。 しかし、既存研究は数百程度の特徴量を使用しているのに対し、我われは 1000 個以上の特徴量から試行を開始した。

将来のオンライン感情認識の観点から、以下の疑問に対する解答を考察する。

• 特徴量選択に対し、大規模な特徴量を与えることは選択される特徴量を良いものにすることができるのか？

• どの程度の自動化が可能なのか？

  • つまり、オンラインシステム上でどの解析ユニットと特徴量が、自動的に計算可能で良い結果を残すのか ？

• 演技あるいは実際の感情を比較した実験はあるが [2] [3] 特徴量セットに対してのものではない。

  • そのため、両方の性質が異なる際にどのような特徴量セットが最適となるのかが分からない。

次章では音声信号からの特徴量抽出の段階を説明する。 続いて、実験を行ったデータベースに関して説明し、最後に実験結果を示す。

2.FEATURE EXTRACTION¶

音声感情認識に対する言語学の領域で一般に使用される韻律的な特徴量はピッチ、エネルギー、 MFCC (Mel Frequency Cepstral Coefficients), ポーズ、 持続時間、 そして話速とフォルマント、声質である(e.g. [1] , [2] , [3] )。 特徴量は与えられた時間的セグメントにおけるこれらの計測値から求められた。 我々のアプローチでは多数の特徴量を計算し、特定のアプリケーションに最も関係するものを選択する。 このコンセプトは他の研究でもよく使われるものだが、本研究ではより網羅的に行う。 100 - 200 の特徴量から選択を行うのではなく、 約 1300 個の特徴量から試行を開始した。

特徴量抽出のプロセスは 3 つのステップに分けられる. セグメントの長さを選択し、それらのセグメントにおける特徴量を計算し、その後最も相関の高い特徴量セットを削減していく。 これらのステップに関して、以後詳細を述べる。

3.DATABASES¶

4.EVALUATION¶

5.CONCLUSIONS¶

結果として、演技音声と実際の音声では要求されるものが全く異なることが示された。 先行研究に対し、演技音声のに対する特徴選択の影響が高く、実際の感情音声に対しより認識が容易になることを発見した。 本文の新規な貢献は演技された感情と自発的な感情に対する選択された特徴量セットの違いを詳細に観察したことである。 演技された感情及び自発感情に対する良い特徴量は重複する部分が少ないことが示された。 演技感情に対してはピッチに関連する特徴量が中心であったのに対し、自発感情ではMFCC (特に低次元) に関連する特徴量が選択された。 これらの違いは自然が感情を認識することを意図する場合、ある手法の初めてのテストであっても演技音声を使用することに意味がないことを示唆する。

最終的に、我われは特徴量の高度な自動化及び、ユニットのセグメント化が不利になることを示すことはできなかったが、 これは我われが選択プロセスの中で与えた特徴量が大き過ぎたことが原因であると考える。

6.REFERENCES¶

Introduction¶

発話者が伝えたいものは構文的な音素や語の言語的ユニットだけではなく、F0 や持続時間、強度や声のトーン、リズムやフレージングを含む発話者の声の音響的変化である。 Eldred and Price [1958] で述べたように、コミュニケーションは ‘何’ を言っているのかのみが含まれているのではなく、’どのように’ 言っているのかもふくまれている。 Fujisaki [2004] では ‘パラ言語情報’ として言及される’どのように’ 関する情報形式には 書き言葉によって伝達される離散的で、カテゴリカルな ‘言語的情報’ と 発話者の性別、年齢、感情などに基本的には左右されない情報である、非言語情報との二つがあることを示唆している。 感情、指標的、社会的、文化的、そして言語的情報を含む様々な種類の音声信号によって伝達される情報の並行通信は Bühler [1934]からある言語学の伝統的命題である。 表情豊かな音声の多重複合性の豊かさに関する様々な研究のレビューとしては、Schroeder [2004], Gobl and Ní Chasaide [2003], Douglas-Cowie et al.[2003] などを参照されたい。

聞き手の認識を評価することを含む、表情豊かな音声の複雑な性質を研究するための1つのアプローチは、 静的なものではなく、何か継続的に時間変化するものである。 感情／効果を形容詞や尺度を使って記述するものではなく、 寧ろ、Cowie et al. [2000] や Schroeder [2004] に記述されているように 例えば感情の大きさや、活性度（能動的、受動的）, 評価(ネガティブ、ポジティブ) などを 聞き手に 例えば ‘Feeltrace’ などの グラフィカルインターフェースを使い評価するように求める。 ある場合にはその後, 主成分分析（PCA）などの統計解析を行い、 ある感情尺度の知覚が、どの音響的（または調音的）特性と関連しているのかを決定する。 これらの線の間で「自由な選択指標」を使うことができる[see, e.g., Campbell and Erickson, 2004]. この研究では、一人の話者から発話された 「えー」(日本語のバックチャンネリング発話) という音声を被験者に提示し, 音声をコンピュータスクリーン上にあるボックスに並べることを求めた. その後、被験者が音を知覚する方法に応じてボックスにラベルを割り当てた。 ついで、PCA を応答の基礎となる関係を決定するために行った。

その他、表情豊かな音声を研究するためによく使われる方法は、 聞き手や実験者により付与される幸せや、悲しみ、怒りなどの感情形容詞ラベルを使用したある特定の感情表現と音響や調音特性を調査することである。 様々な研究は音響的変化は特定の感情／効果に影響を及ぼすことを報告している[see e.g., review articles by Scherer,2003; Gobl and Ní Chasaide, 2003; Erickson, 2005].

明瞭度も効果に影響する。 例えば、Maekawa et al. [1999] や Erickson et al. [2000] では 発話者が異なったパラ言語的状況、例えば、疑いや賞賛、怒りなどで同じ発話を発話した もらった場合舌や顎の位置は変化することを明らかにした。

特定の感情を要求されて話し手が発話するような演技音声と自発的感情表現を区別することが重要である。 演技された感情は特定の影響を聞き手に伝えようと声優は口頭表現を制御するため、 演技音声は発話者により、これらの感情が経験されるものとは別のカテゴリになる場合が多い。 例えば、深い悲しみを表現するためにある話者は発話中に泣き始めるかもしれない。 泣くことは言語的メッセージの一部ではない。 一般に発話者は泣くことを制御できない。 これはしばしば、発話者の意図に拘わらずおき、生理的感情的に脳内で誘発された変化の結果である. Brown et al. [1993] のような研究では悲しみや高揚の結果としてのホルモン変化を示している。 悲しみや高揚など特定の感情による発話者の生物学的状態の変化が 喉頭と前喉頭関節が変化することもありうる。 Erickson et al. [2003a, b, 2004a-c] による予備調査では、演技音声と自発的で表情豊かな音声は舌や顎の明瞭度が異なるパターンを持ち、発声されていることを示した。 Schroeder et al. [1998] では、面白がる音声の聴覚-視覚的特性の調査を行い、 被験者が模倣された感情(驚きの表情)と本物の感情を区別して知覚できることを発見している。

発話者に同時に発話を行うことを強制する、例えば’泣く’といった発話者による特定の種類の強い感情表現は 感情と発話の間の可能な生理的な接続を考えるための窓口が開くため、 どの程度の感情と音声は関連をしており、どの程度に強い感情は、言語的タスクを妨害または高めることができるのかという研究をすることは 知覚と発話の情報の観点から 面白いものである。

Methods¶

Articulatory Analysis Method¶

We examined the movement of the EMA receiver coils attached to the (1) lower incisor(mandible) (2) upper lip, (3) lower lip, and (4) receiver coils (T1, T2, T3, T4) attached along the longi-tudinal sulcus of the speaker’s tongue. The positions of the transmitter coils determine thecoordinate system [Kaburagi and Honda, 1997] with the origin positioned slightly in front of and belowthe chin. All EMA values are positive, with increasingly positive y-values indicating increasingly raisedjaw or tongue position, and increasingly positive x-values, increasingly retracted jaw or tongue position.The coordinates of the American English speaker’s first recording were transformed by 5 degrees,which was the measured difference between the coordinates in the two sessions [Erickson et al., 2003b].

Articulatory measurements were made for the x-y coil positions for the upper and lower lip (UL,LL), for the mandible (J), and the tongue (T1, T2, T3, T4) at the time of maximum jaw opening for theutterance, using a MATLAB-based analysis program. Note that T4 recordings were not reliable foreither of the speakers, and T3 was not reliable for the Japanese speaker because of coil-tracking prob-lems. For the American English utterances, articulatory as well as acoustic measurements were madeat the time of maximum jaw opening for leave. For the Japanese utterances, articulatory measure-ments were made at the time of maximum jaw opening during the second mora in the utterance kara.

• As seen from the video images, for the Japanese speaker, the manifestation of crying was tears running down thecheeks; for the American speaker, the face was contorted in a sobbing posture with jaw open and eyebrows knittogether, as well as tears.

A sample of a data file is shown for Japanese in figure 1. Notice there are two distinct jaw openings,one for each mora. However, for two of the utterances (one IE and one R), there was only one jawopening, which occurred at the plosion release for /k/.

Fig. 1. Sample articulatory data of kara from utterance 81 (E). The bottom trace is the acoustic wave-form. Vertical lines indicate lowest Jy position in each mora. Articulatory measurements made duringthe second mora were used in the analysis. The x-axis shows time in seconds. The values of the tickmarkings for the y-axes are as follows: Jx 7.85-7.80 mm; Jy 12.8-12.6 mm; ULx 5.9-5.85 mm; ULy15.2-15.15 mm; LLx 6.45-6.3 mm; LLy 13.6-13.3 mm; T1x 9.2-8.8 mm; T1y 14.8-14.0 mm; T2x10.2-9.8 mm; T2y 20-14.5 mm.

Results¶

Perception Test Results¶

For both American English and Japanese, listeners ranked emotional and imitatedemotional speech as sadder than read speech, and in the case of American English,emotional and imitated emotional speech as sadder than imitated intonation speech, ascan be seen in figure 2.

One-way ANOVA of the perceptual ratings (mean for all the listeners) with theutterance conditions (4 levels: E, IE, II, R for American English and 3 levels: E, IE, R forJapanese) as an explanatory variable found significant main effects for both languages(American English, F(3,12) = 351.328, p < 0.000; Japanese, F(2,11) = 14.704,p < 0.001). The Bonferroni pairwise comparisons revealed that perceptual rating ofsadness is significantly higher for E and IE compared to II and R for American English(p < 0.000), and for IE compared to R for Japanese (p = 0.001). The finding that imitated intonation in American English was not rated as sad as spontaneous or imitated various studies about the multicomplex richness of expressive speech, see e.g.,Schroeder [2004], Gobl and Ní Chasaide [2003] and Douglas-Cowie et al. [2003].

Fig. 2. Ratings by American listeners of the perceived degree of sadness of 16 utterances of leave(left side) and ratings by Japanese listeners of perceived degree of sadness of 14 utterances of kara(right side). The error bars indicate standard deviation of p = 0.68.

Imitated expressions of sadness (IE) were actually rated as slightly sadder thanspontaneous expressions of sadness (E) for both American English and Japanese.However, the differences were not significant. A larger sample size is necessary to fur-ther explore this.

Acoustic Analysis Results¶

Fig.3. Results of acoustic measurements, both for American English leave (left side) and Japanese kara (right side) for each of the utterance conditions. From top to bottom, duration, F0, F1, AQ (for American English) and spectral slope (for Japanese). The error bars indicate standard deviation of p = 0.68.

Figure 3 shows the results of the acoustic analysis of American English and Japanese speech. The averaged acoustic values are shown in tables 1A and 2A in the Appendix.

Figure 3 shows that F0, F1 and voice quality (but not duration) of imitated sadspeech are similar to those for spontaneous sad speech. One-way ANOVA of theacoustic measures (F0, F1, voice quality, and duration) with the utterance conditions (E,IE, II, R for American English and E, IE, R for Japanese) as an explanatory variablefound main effects for duration, F0 and AQ for American English and for duration andF0 for Japanese (table 3A in the Appendix). Bonferroni pairwise comparisons found nosignificant differences between the E and IE conditions for F0, duration and voice quality for either the American English or Japanese. Both speakers showed a higher F0 forsad and imitated sad speech than for read speech (and imitated intonation speech). Bonferroni pairwise comparisons found F0 was significantly higher for E and IE com-pared to II and R for American English (p < 0.000), and for E and IE compared to R(p = 0.013 and p = 0.038, respectively) for Japanese. The finding of high F0 (whichwas clearly audible) is different from that usually reported for sad quiet speech, butsimilar to what was found for the active grieving seen in Russian laments.

F1 values were similar for spontaneous and imitated sad compared to those for read(and imitated intonation) speech: for the American English speaker (for the vowel /i/), sadness suggests that intonation itself is not sufficient to convey sadness; as discussed in’Results of Pearson Correlation with Ratings of Sadness and Accoustic/ArticulatoryMeasures’ below, voice quality and F0 height are salient characteristics of sadness.

F1 values for sad and imitated sad speech were higher, and for the Japanese speaker(for the vowel /a/), they were lower. These differences were not significant. However,the graphs suggest a tendency for the /i/ and /a/ vowels to centralize in sad speech; thisneeds further investigation. The finding of low F1 for sad and imitated sad utterancesfor the Japanese speaker is reminiscent of the finding of lowered F1 values for disap-pointment (also for the vowel /a/) reported by Maekawa et al. [1999].

In addition, both speakers showed changes in voice quality for the sad and imi-tated sad speech, compared with the read (and imitated intonation) speech. For theAmerican English speaker, both spontaneous and imitated sad speech have a low AQ.Bonferroni pairwise comparisons found AQ significantly lower for E and IE comparedto II (E < II: p = 0.018; IE < II: p < 0.000), and for the Japanese speaker, a steepspectral tilt (but no significant differences).

The finding of low AQ for the American English spontaneous sad utterances is dif-ferent from the high AQ previously reported for sad, quiet breathy speech. It may be thatlow AQ is seen in active grieving because crying would probably involve muscular ten-sion, including tension of the vocal folds. When vocal folds are tense/pressed, the dura-tion of vocal fold closure is large, and this would lower AQ.

Fig. 4. Sample acoustic wave forms for Japanese kara. The top panel is sad speech (E), next is imitatedsad speech (IE), and the bottom is read (R) speech. Each tick marking on the time axis indicates 100 ms.

As for the Japanese, spontaneous sad speech was perceived by the authors to bebreathy-voiced and the imitated sad, creaky-voiced, and this can be seen in the acousticwaveforms in figure 4. The spontaneous sad speech in the top panel of the figure shows relatively smooth phonation for breathy voice [similar to that shown for breathy voice by Ishii, 2004]; the imitated sad speech in the middle panel shows irregular and sporadic phonation, typical of creaky voice [see also, e.g., Ishii, 2004, as well as e.g., Redi and Shattuck-Hufnagel, 2001], and the read speech in the bottom panel shows regular pulsing with clear vowel-formant characteristics, though some creakiness towards the end.

A comment about spectral tilt and breathiness/creaky voice: That breathy utterances have steep spectral slopes is relatively well known [e.g., Klatt and Klatt, 1990; Johnson, 2003], as is also that sad utterances tend to be breathy [i.e., Mohktari and Campbell, 2002]. Creaky utterances at low F 0 are not characterized by steep spectral tilt [e.g., Klatt and Klatt, 1990; Johnson, 2003]. However, in the Japanese data we obtained, we see a steep spectral tilt associated with the creaky voice of the imitated sadness. It may be that depending on the value of F 0 , the spectral slope of the creaky utterances changes, so that at high F 0 , creaky utterances have very steep spectral slopes. As Gerratt and Kreiman [2001] argue, characterization of variation in nonmodal phonation is not straightforward due to complications of both taxonomy and methods. Phonetic characterization of voice quality changes associated with emotions in speech has not yet been done; however, see e.g., Gobl and Ní Chasaide [2003] for their summary of voice quality analysis of expressive speech.

The averaged durations of the imitated sad speech for the American Englishspeaker were generally longer than those of the spontaneous sad speech, as well as theother conditions. However, the only significant difference was for IE compared to II(p = 0.007). Perhaps the reason the imitated sad speech was longer than the sponta-neous sad speech was because the speaker was expecting sad speech to be slow, andmay have inadvertently allowed this expectation to influence her production.

For the Japanese speaker, the spontaneous and imitated sad utterances tended to belonger than the read utterances and significantly longer for E compared to R(p = 0.023), which is consistent with what is reported in the literature for sad speech.