Inner 'n_bytes >= 0' will always be true.

utf8_nbytes() is only used in dictionary.c. Others are moved to common.c

./src/espeak-ng -v ar --path=$PWD 5777

would access out of the vowel_stress array, because GetVowelStress, in the
phcode == phonSYLLABIC case, only increments count if stress is 0.  This
code seems fishy, and is not coherent with the loop in SetWordStress.  I
made it similar to the other case above, thus fixing the out-of-bound
access.

This is modifiying the stress in the tn test, I don't know whether this
is expected.

utf8_in2, when working in backward mode, is assuming that we are giving
the address of the last byte of the previous character (see all other
calls to utf8_in2). Otherwise, utf8_in2 returns the size of the current
multibyte character instead of that of the previous multibyte character.

With emojis, phoneme strings can be very long

IsLetterGroup looks for heading \0, so the padding also needs to include
an \0.

So valgrind can potentially catch e.g. missing \0.

So we know they are just tables, get protected from overflows, and get shared between processes

Some rules test against character not being of a certain type. That may
match with the \0 end-of-text marker, and thus actually step over
it and let MatchRule continue with uninitialized data after it, leading
to potential random behavior.

This commits fixes it by making sure that we don't read past that \0.

This seems to be changing the pronunciation of "capitals" from k'apIt@Lz to
k'apIt,alz, I don't know why, I guess the rule for it was actually
bogus?

Special characters such as N, S1, etc. are not actually eating
characters. Their treatment should thus *not* update pre_ptr and post_ptr,
otherwise those would underflow/overflow, e.g. in the case

@) s (_NS1  [z]

this would overflow. This for instance noticeable with the memory sanitizer:

ESPEAK_DATA_PATH=$PWD ./src/espeak-ng -qX "capitals"
Translate 'capitals'
  1	c        [k]

  1	a        [a]

  1	p        [p]

  1	i        [I]

  1	t        [t]

  1	a        [a]

  1	l        [l]
 20	l (C     [l]

==2837201==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f7f4422744b in utf8_in2 /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:281:2
    #1 0x7f7f442281bc in utf8_in /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:332:9
    #2 0x7f7f440e0d31 in MatchRule /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/dictionary.c:1767:21
    #3 0x7f7f440d937f in TranslateRules /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/dictionary.c:2320:6
    #4 0x7f7f44230e5f in TranslateWord3 /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:733:15
    #5 0x7f7f44229844 in TranslateWord /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:1100:14
    #6 0x7f7f44256e50 in TranslateWord2 /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:1361:11
    #7 0x7f7f4424d6cc in TranslateClause /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:2623:17
    #8 0x7f7f44213359 in SpeakNextClause /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/synthesize.c:1569:2
    #9 0x7f7f441a9f56 in Synthesize /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/speech.c:457:2
    #10 0x7f7f441a9023 in sync_espeak_Synth /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/speech.c:570:29
    #11 0x7f7f441ad59f in espeak_ng_Synthesize /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/speech.c:678:10
    #12 0x7f7f4410b3f4 in espeak_Synth /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/espeak_api.c:90:32
    #13 0x4a8be3 in main /home/samy/brl/speech/espeak-ng-git/src/espeak-ng.c:691:3
    #14 0x7f7f43a2e7fc in __libc_start_main csu/../csu/libc-start.c:332:16
    #15 0x421449 in _start (/home/samy/ens/projet/1/speech/espeak-ng-git/src/.libs/espeak-ng+0x421449)

  Uninitialized value was created by an allocation of 'sbuf' in the stack frame of function 'TranslateClause'
    #0 0x7f7f4423a1f0 in TranslateClause /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:1941

While trying to match _NS1, MatchRule is overflowing the buffer.

It happens that this had not usually posed problem because rules usually
have these non-eating special characters last in the rule and thus it wasn't
mattering that post_ptr is pointing outside valid text.

When matching a double-letter whose encoding needs multiple bytes, to
properly skip the double-letter we have to skip these bytes, like other
rules.

Some rules test against character not being of a certain type. That may
match with the \0 beginning-of-text marker, and thus actually step over
it and let MatchRule continue with uninitialized data before it, leading
to potential random behavior.

This commits fixes it by making sure that we don't read before that \0.

LookupDict2 looks forward in the wtab array, it should still stop at its
end. Otherwise the memory sanitizer reports this:

testing en A. B C, D. E: F.
==65960==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7ff9d7ef0de8 in LookupDict2 /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/dictionary.c:2676:11
    #1 0x7ff9d7eec2ec in LookupDictList /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/dictionary.c:2899:10
    #2 0x7ff9d802860a in TranslateWord3 /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:588:12
    #3 0x7ff9d80249d4 in TranslateWord /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:1100:14
    #4 0x7ff9d8051fe0 in TranslateWord2 /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:1361:11
    #5 0x7ff9d804885c in TranslateClause /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:2623:17
    #6 0x7ff9d800e4e9 in SpeakNextClause /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/synthesize.c:1569:2
    #7 0x7ff9d7fa50e6 in Synthesize /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/speech.c:457:2
    #8 0x7ff9d7fa41b3 in sync_espeak_Synth /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/speech.c:570:29
    #9 0x7ff9d7fa872f in espeak_ng_Synthesize /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/speech.c:678:10
    #10 0x7ff9d7f06584 in espeak_Synth /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/espeak_api.c:90:32
    #11 0x4a8be3 in main /home/samy/brl/speech/espeak-ng-git/src/espeak-ng.c:691:3
    #12 0x7ff9d78297fc in __libc_start_main csu/../csu/libc-start.c:332:16
    #13 0x421449 in _start (/home/samy/ens/projet/1/speech/espeak-ng-git/src/.libs/espeak-ng+0x421449)

  Uninitialized value was created by an allocation of 'words' in the stack frame of function 'TranslateClause'
    #0 0x7ff9d8035380 in TranslateClause /home/samy/brl/speech/espeak-ng-git/src/libespeak-ng/translate.c:1941

Strictly speaking, we are not supposed to use memcmp to compare strings
since we are not supposed to read beyond \0, which memcmp is supposed to
potentially do. Sanitizers would warn about it, and using strncmp happens to
provide the proper semantic while being not really slower, so better
just use them.

strlen(p) may be arbitrarily long, that would underflow the word, for
instance:

testing fr Latn
=================================================================
==3741805==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd733c1329 at pc 0x7ff5ffbad2de bp 0x7ffd733bf310 sp 0x7ffd733bf308
READ of size 1 at 0x7ffd733c1329 thread T0
    #0 0x7ff5ffbad2dd in IsLetterGroup src/libespeak-ng/dictionary.c:714
    #1 0x7ff5ffbbe425 in MatchRule src/libespeak-ng/dictionary.c:1979
    #2 0x7ff5ffbc09e9 in TranslateRules src/libespeak-ng/dictionary.c:2301
    #3 0x7ff5ffc26656 in TranslateWord3 src/libespeak-ng/translate.c:733
    #4 0x7ff5ffc2a10b in TranslateWord src/libespeak-ng/translate.c:1100
    #5 0x7ff5ffc2bef2 in TranslateWord2 src/libespeak-ng/translate.c:1361
    #6 0x7ff5ffc374e2 in TranslateClause src/libespeak-ng/translate.c:2623
    #7 0x7ff5ffc1d010 in SpeakNextClause src/libespeak-ng/synthesize.c:1569
    #8 0x7ff5ffbfbd46 in Synthesize src/libespeak-ng/speech.c:492
    #9 0x7ff5ffbfd52a in sync_espeak_Synth src/libespeak-ng/speech.c:570
    #10 0x7ff5ffbfdd1f in espeak_ng_Synthesize src/libespeak-ng/speech.c:678
    #11 0x7ff5ffbc72fd in espeak_Synth src/libespeak-ng/espeak_api.c:90
    #12 0x5627511a3137 in main src/espeak-ng.c:691
    #13 0x7ff5fee557fc in __libc_start_main ../csu/libc-start.c:332
    #14 0x5627511a0569 in _start (/home/samy/ens/projet/1/speech/espeak-ng-git/src/.libs/espeak-ng+0x6569)

Address 0x7ffd733c1329 is located in stack of thread T0 at offset 1177 in frame
    #0 0x7ff5ffc2f760 in TranslateClause src/libespeak-ng/translate.c:1941

  This frame has 16 object(s):
    [48, 52) 'cc' (line 1944)
    [64, 68) 'source_index' (line 1945)
    [80, 84) 'prev_in' (line 1948)
    [96, 100) 'prev_out' (line 1949)
    [112, 116) 'next_in' (line 1952)
    [128, 132) 'char_inserted' (line 1954)
    [144, 148) 'word_flags' (line 1963)
    [160, 164) 'charix_top' (line 1975)
    [176, 180) 'tone' (line 1985)
    [192, 196) 'next2_in' (line 2294)
    [208, 212) 'c_temp' (line 2518)
    [224, 374) 'number_buf' (line 2522)
    [448, 1048) 'num_wtab' (line 2523)
    [1184, 1984) 'sbuf' (line 1982) <== Memory access at offset 1177 underflows this variable
    [2112, 3720) 'charix' (line 1977)
    [3856, 7456) 'words' (line 1978)

sbuf is however properly '\0'-header, so we can make IsLetterGroup
carefully walk back in the word and issue a mismatch if it walks back
too much.

Fixes #1108

strlen(p) may be arbitrarily long, that would underflow the word, for
instance:

testing fr Latn
=================================================================
==3741805==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd733c1329 at pc 0x7ff5ffbad2de bp 0x7ffd733bf310 sp 0x7ffd733bf308
READ of size 1 at 0x7ffd733c1329 thread T0
    #0 0x7ff5ffbad2dd in IsLetterGroup src/libespeak-ng/dictionary.c:714
    #1 0x7ff5ffbbe425 in MatchRule src/libespeak-ng/dictionary.c:1979
    #2 0x7ff5ffbc09e9 in TranslateRules src/libespeak-ng/dictionary.c:2301
    #3 0x7ff5ffc26656 in TranslateWord3 src/libespeak-ng/translate.c:733
    #4 0x7ff5ffc2a10b in TranslateWord src/libespeak-ng/translate.c:1100
    #5 0x7ff5ffc2bef2 in TranslateWord2 src/libespeak-ng/translate.c:1361
    #6 0x7ff5ffc374e2 in TranslateClause src/libespeak-ng/translate.c:2623
    #7 0x7ff5ffc1d010 in SpeakNextClause src/libespeak-ng/synthesize.c:1569
    #8 0x7ff5ffbfbd46 in Synthesize src/libespeak-ng/speech.c:492
    #9 0x7ff5ffbfd52a in sync_espeak_Synth src/libespeak-ng/speech.c:570
    #10 0x7ff5ffbfdd1f in espeak_ng_Synthesize src/libespeak-ng/speech.c:678
    #11 0x7ff5ffbc72fd in espeak_Synth src/libespeak-ng/espeak_api.c:90
    #12 0x5627511a3137 in main src/espeak-ng.c:691
    #13 0x7ff5fee557fc in __libc_start_main ../csu/libc-start.c:332
    #14 0x5627511a0569 in _start (/home/samy/ens/projet/1/speech/espeak-ng-git/src/.libs/espeak-ng+0x6569)

Address 0x7ffd733c1329 is located in stack of thread T0 at offset 1177 in frame
    #0 0x7ff5ffc2f760 in TranslateClause src/libespeak-ng/translate.c:1941

  This frame has 16 object(s):
    [48, 52) 'cc' (line 1944)
    [64, 68) 'source_index' (line 1945)
    [80, 84) 'prev_in' (line 1948)
    [96, 100) 'prev_out' (line 1949)
    [112, 116) 'next_in' (line 1952)
    [128, 132) 'char_inserted' (line 1954)
    [144, 148) 'word_flags' (line 1963)
    [160, 164) 'charix_top' (line 1975)
    [176, 180) 'tone' (line 1985)
    [192, 196) 'next2_in' (line 2294)
    [208, 212) 'c_temp' (line 2518)
    [224, 374) 'number_buf' (line 2522)
    [448, 1048) 'num_wtab' (line 2523)
    [1184, 1984) 'sbuf' (line 1982) <== Memory access at offset 1177 underflows this variable
    [2112, 3720) 'charix' (line 1977)
    [3856, 7456) 'words' (line 1978)

sbuf is however properly '\0'-header, so we can make IsLetterGroup
carefully walk back in the word and issue a mismatch if it walks back
too much.

Fixes #1108

pre_ptr is already one byte before the current letter, so we do not want
to subtract 1 again. Otherwise this would for instance underflow word_iz
of addPluralSuffixes.

pre_ptr is already one byte before the current letter, so we do not want
to subtract 1 again. Otherwise this would for instance underflow word_iz
of addPluralSuffixes.

This will make it easier to set a default voice other than
English. This is important for cases when a language will fall back to
the default voice.

Some references to L('e', 'n') still need to be changed.

There are nolanguages that use stress_rule = 10.

The #defines should be renamed to better convey their meaning.

The flag is never set anywhere so the if-clause will always evaluate to false.

- param2[] is only used to set a second value to LOPT_BRACKET_PAUSE. It is simpler
to have two values in param[] instead. This simplifies the codebase.

- Instead of setting "option bracket X Y" in language files, use
keywords "brackets X" and "bracketsAnnounced Y" instead to follow the
naming convention of other keywords.

- Add missing documentation to docs/voices.md.

Going through files in src/libespeak-ng/, include-what-you-use removed a
few unnecessary includes and included explanations on why a certain
header should be included. This makes tracking globals and dependencies easier.

Running the codebase through IWYU should be repeated after each major
code restIncludes to standard c library weren't checked to avoid
breaking builds with other platforms.

See https://github.com/include-what-you-use/include-what-you-use

The replacement tests for bs, hr, and sr are no longer marked as
broken as they work using the old code. The mk tests keep the
broken annotation, as they don't work in the old code either.

This reverts commit 801a8d197c.
This reverts commit 64d5701e5e.
This reverts commit 3b51ebf617.
This reverts commit 1fd235d2c0.
This reverts commit 9f0667de86.

This is a fix for https://github.com/nvaccess/nvda/issues/7740.

With the addition of emoji support, dictionary entries can now be
longer than 128 bytes. This fix makes sure the character is
interpreted as an unsigned byte so it does not treat long entries
as having a negative offset.

Treating the offset as a signed byte (like in the previous code)
could cause the hash chain search to loop indefinitely when
processing certain input, like the Tamil characters in the NVDA
issue noted above that is added as a test case to translate.test.