- fix a buffer overflow in ucd_tolower leading to failure when
compiling with address sanitizer
- force the use of C++ compiler for espeak-ng
- adding a malloc to have a null-terminated string in the fuzz target
- setting (but not overwriting) ESPEAK_DATA_PATH environment
variable inside the fuzz target

With the previous calculation, it was possible for the buffer to be
a byte or two too small to hold all of the data.

espeak-ng --compile-phonemes generates phontab, but during the
generation it starts by reading the old phontab.  If works just fine,
apart from an error message, if this doesn't exist but if it does exist
it is not clear whether it might end up using old or bad data (e.g.
possibly crashing if phontab is corrupted making it very non-obvious how
to fix the problem...)

The same applies to en_dict as a result of espeak-ng defaulting to the
voice 'en'; again the phontab build works if en_dict is absent but a
spurious message is output.  It's not clear what would happen if someone
tried to build a reduced espeak-ng configuration with just one (non
'en') language.

The fix is to change espeak_ng_CompilePhonemeDataPath to call LoadVoice
with a new flag to indicate the voice structure is just required to
compile phoneme data (I changed the error message as well to make it
clear where it comes from - there are two identical error messages, the
other one is in voices.c)

Within LoadVoice the flag causes LoadVoice to not set a default language
or voicename and to skip loading the old phontab or dictionary (both of
which fail in a git clean -dfx build).

I checked the result by doing a compile from-scratch build sequence with
and without the patch, i.e. I did:

git clean -dfx
./autogen.sh
./configure --prefix=a-test-directory
make
make install

Then I compared the two copies of "a-test-directory" as well as the log
file output:

$ diff -r logs/*
diff -r logs/original/make.log logs/patched/make.log
175,176d174
< Unknown phoneme table: 'en'
< Can't read dictionary file:
'/home/jbowler/src/espeak-ng/master/espeak-ng-data/en_dict'

$ diff --no-dereference -r local.original local.patched
Binary files local.original/bin/speak-ng and local.patched/bin/speak-ng differ
Binary files local.original/lib/libespeak-ng.a and local.patched/lib/libespeak-ng.a differ
Binary files local.original/lib/libespeak-ng.so.1.1.51 and local.patched/lib/libespeak-ng.so.1.1.51 differ

So the change has not altered phontab or any of the other generated
files, only the binaries as would be expected.  (Note that I am
comparing truely clean builds; what you get if you download the source
and build from scratch.  This is the standard approach for people who
build for distribution.)

Signed-off-by: John Bowler <[email protected]>

The only way (in Dec. 2019) to get espeak_ListVoices to actually list
all the voices, including variants and mbrola voices used to be to pass
a non-null voice selector with a null language pointer.  This still
seems to be supported in SetVoiceScores, which still checks for a null
language (see the check right at the top), however new code has been added
for a pseudo voice "all" lower down which seems to do the same thing but
crashes with null.  This change allows "old" code to continue working.

Signed-off-by: John Bowler <[email protected]>

93d3c67df accidentally changes number handling for finnish.

D_FRACTION2 seems to match finnish number standards. It causes the
decimals to be read as numbers instead of individual digits: 12,12 is
read "twelve point twelve" instead of "twelve point one two".

It is undocumented so it might cause regressions as well.

Co-authored-by: Sebastien Le Maguer <[email protected]>

When set in a language file, a period followed by a lower case letter is
detected as end of sentence. Normal behavior is to require a capital
letter.

All sound file information goes to soundicon_tab. The contents of the files are not
loaded to memory before they are needed for synthesis.

This removes N_SOUNDICON_SLOTS that divided soundicon_tab into two
parts for unknown reasons.

TODO: LoadConfig() is currently undocumented. The logic should be
moved to LoadVoice() and documented.

Soundicons are used for external audio with SSML <audio> tag and for
replacing punctuation names with sound files in LoadConfig().

Currently there's a bug wih soundicon slots: if both LoadConfig and
<audio> are used, the punctuation reserves all slots and no sound from
<audio> is played.

The SelectPhonemeTable(voice->phoneme_tab_ix) has no effect for tests.
Passing other values here breaks tests. That means that the relevant
calls SelectPhonemeTable() are already made in other functions.

voice->phoneme_tab_ix is set in LoadVoice().

wavefile_ix is always equal to fmt_params->wav_addr.

wavefile_amp is only used once  so it can be set right
before calling DoSample2().

speed.fast_settings[] was introduced in 7b2eecc724 to be used with
Sonic. Looks like after 7b2eecc724 the code related to everything
except fast_settings[0] was removed.

Might be that speed.fast_settings[0] is also irrelevant and could be
removed.

It is set but never used.

See commit message for 23a4d88f.

This commit fixes cmn and yue.
CalcPitches_Tone() now accepts cmn for translator_name.
SelectTranslator() now has a case for yue instead of zhy.

This reverts commit 23a4d88f6b.

Breaks tests for cmn.

Option "language <name>"already causes SelectTranslator(<name>) to be
called. Having two options to do almost the same thing is unnecessary
and confusing.

In the long term, all options from SelectTranslator() should have a
switch case in LoadVoice() so they are user configurable (see #218). If
needed, a new option (maybe called "LoadOptions") could be added to load
an existing voice or language file.

Changes language configuration files for: hak, cmn, yue, ltg, ms, mb-ma1.
No changes to users.

The language is not supported and apparently never has been.

docs: add details about number flags to the documentation.

It's clearly intended to be enabled by default:
- it's defined as default behaviour translate.h (NUM_DEFAULT)
- tr_languages.c sets many default values related to number processing
  that have no meaning unless langopts.numbers == 1.

It is also a more sensible default since most languages will want to
have number processing on. This makes adding new languages easier
because  adding an entry to tr_languages.c is unnecessary.

A negative side effect is that languages with partial number defines
might experience bugs when reading undefined numbers. This is a bug and
should be fixed.

This will have the side effect of enabling number processing for
languages that currently have it disabled. However, there shouldn't be
any.

Here's a way to check affected languages:
for voice in $(ESPEAK_DATA_PATH=`pwd` LD_LIBRARY_PATH=src:${LD_LIBRARY_PATH}
src/espeak-ng --voices | grep -v Languages | awk '{print $2}'); do
OUTPUT=$(ESPEAK_DATA_PATH=`pwd` LD_LIBRARY_PATH=src:${LD_LIBRARY_PATH}
src/espeak-ng -qx -v $voice "1 - 2 - 3 - 12 - 123") && echo "$voice:
$OUTPUT" ; done

These voices clearly benefit from enabling numbers (they already have
number rules in *_list):
ba, cmn (zh), hak, haw, ja, kok, nb, nci

Some languages are missing some definitions (like _12) in _list files.
It causes the program to skip some numbers.
Numbering needs to be turned off explicitly for:
jbo, mi, my, piqd, py, qu, quc, th, uz

Languages with no number rules at all:
chr, cv, he, nog, tk, ug

Espeak-ng uses bitrate 16 internally. If the input inside <audio> has a
different bitrate the audio will not play correctly unless converted to
16 bits.

Closes #885.

It is supposed to be used with SSML for reading < (&lt;) and & (&amp;)
when invoking espeak-ng --punct -m.

However, it looks like the code is dead since removing it doesn't fail
the ssml reference test.

Instead of passing a temporary variable current_voice_id to SSML
processing logic and then copying the value to voice_change, pass
voice_change directly. GetVoiceAttributes() both sets voice_change and
retuns CLAUSE_TYPE_VOICE_CHANGE that is used by TranslateClause() for
error checking.

tests/ssml/language-switch.ssml covers the intended behavior of language
switching and will fail when either voice_change or CLAUSE_TYPE_VOICE_CHANGE
is not set correctly.

This approach reduces the amount of nested if statements and doesn't
require the temporary variable int c_next_2.

A test already exists for this case in tests/translate.test

- V_ALPHABET2 is a leftover from 2f8f125.
- langopts assignment is unnecessary since it's not used later in the
function.

It is set but never used.

This adds a corresponding header file for mnemonics.c that was introduced in d5d93be.

- param2[] is only used to set a second value to LOPT_BRACKET_PAUSE. It is simpler
to have two values in param[] instead. This simplifies the codebase.

- Instead of setting "option bracket X Y" in language files, use
keywords "brackets X" and "bracketsAnnounced Y" instead to follow the
naming convention of other keywords.

- Add missing documentation to docs/voices.md.

There is no code related to it.

It is set but never used.

This reverts commit f0c345eb6a.

Fixes #824.

translate.h gives two settings to LOPT_CAPS_IN_WORD, but only one of them is actually ever used.

It is enough to have the logic in dictsource/jbo_*.

Thanks to xunsku @ irc.freenode.org

It's unclear why c2 needs to be set after an entity reference.

ProcessSsmlTag()

Note the line in readclause.c:
if ((c2 == '/') || iswalpha(c2) || c2 == '!' || c2 == '?') {

It might be enough to pass everything to ProcessSsmlTag. What are the
cases that are skipped because of this?

This is a bit slower since we don't pass n_xml_buf as an argument but
rather get it with a call to wcslen. It is much cleaner though, since
the name ProcessSsmlTag() implies that all processing should be done
there.