How to make autocompletion work on IME

[johnsonj]

Let me discuss this theme separately.
I have finally found out why autocompletion works randomly.
Through lots of trial and error, I found the right place where notify()
sits.

the patch follows


---------------------------------------------------------------------------------
void ScintillaBase::AddCharUTF(char *s, unsigned int len, bool treatAsDBCS)
{
bool isFillUp = ac.Active() && ac.IsFillUpChar(*s);
if (!isFillUp) {
Editor::AddCharUTF(s, len, treatAsDBCS);
}
if (ac.Active()) {
AutoCompleteCharacterAdded(s[0]);
// For fill ups add the character after the autocompletion has
// triggered so containers see the key so can display a calltip.
bool isFillUp = ac.IsFillUpChar(*s);
if (isFillUp) {
Editor::AddCharUTF(s, len, treatAsDBCS);
}
}
// moved from Editor::AddCharUTF
if (treatAsDBCS) {
NotifyChar((static_cast<unsigned char>(s[0]) << 8) |
static_cast<unsigned char>(s[1]));
} else if (len > 0) {
int byte = static_cast<unsigned char>(s[0]);
if ((byte < 0xC0) || (1 == len)) {
// Handles UTF-8 characters between 0x01 and 0x7F and single
byte
// characters when not in UTF-8 mode.
// Also treats \0 and naked trail bytes 0x80 to 0xBF as valid
// characters representing themselves.
} else {
// Unroll 1 to 3 byte UTF-8 sequences. See reference data at:
// http://www.cl.cam.ac.uk/~mgk25/unicode.html
// http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt
if (byte < 0xE0) {
int byte2 = static_cast<unsigned char>(s[1]);
if ((byte2 & 0xC0) == 0x80) {
// Two-byte-character lead-byte followed by a
trail-byte.
byte = (((byte & 0x1F) << 6) | (byte2 & 0x3F));
}
// A two-byte-character lead-byte not followed by trail-byte
// represents itself.
} else if (byte < 0xF0) {
int byte2 = static_cast<unsigned char>(s[1]);
int byte3 = static_cast<unsigned char>(s[2]);
if (((byte2 & 0xC0) == 0x80) && ((byte3 & 0xC0) == 0x80)) {
// Three-byte-character lead byte followed by two trail
bytes.
byte = (((byte & 0x0F) << 12) | ((byte2 & 0x3F) << 6) |
(byte3 & 0x3F));
}
// A three-byte-character lead-byte not followed by two
trail-bytes
// represents itself.
}
}
NotifyChar(byte);
}
}
---------------------------------------------------------------------------------------------------
IME do not need to have notifychar();
It works well in both utf8 and dbcs on win32.
On gtk not tested yet.
I am very pleased.
Just take a review.

Thanks

[johnsonj]

treatAsDBCS works good in both utf8 and dbcs.

unicode still does not work.
Is there any need of unicode?
I do not understand.

[johnsonj]

Let put it simplest for helping understanding.
It also works.
======================================================
void ScintillaBase::AddCharUTF(char *s, unsigned int len, bool treatAsDBCS)
{
bool isFillUp = ac.Active() && ac.IsFillUpChar(*s);
if (!isFillUp) {
Editor::AddCharUTF(s, len, treatAsDBCS);
}
if (ac.Active()) {
AutoCompleteCharacterAdded(s[
0]);
// For fill ups add the character after the autocompletion has
// triggered so containers see the key so can display a calltip.
bool isFillUp = ac.IsFillUpChar(*s);
if (isFillUp) {
Editor::AddCharUTF(s, len, treatAsDBCS);
}
}
// moved from Editor::AddCharUTF
NotifyChar((static_cast<unsigned char>(s[0]));
==========================================================

[Neil Hodgson]

Post by johnsonj
Let put it simplest for helping understanding.
It also works.
...
bool isFillUp = ac.IsFillUpChar(*s);

Changing the above line should not have any effect since the set of fill up characters should not change.

Post by johnsonj
...
// moved from Editor::AddCharUTF
NotifyChar((static_cast<unsigned char>(s[0]));

This still appears to be just compensating for SciTE not behaving sensibly with characters >255. It would be better to fix SciTE and leave the meaning of the character notification unchanged as changing it will break other applications.

As a first pass at fixing SciTE, find the handler for SCN_CHARADDED and change the call to CharAdded to

if (notification->ch < 128)
CharAdded(static_cast<char>(notification->ch));

Neil

[johnsonj]

""" This still appears to be just compensating for SciTE not behaving
sensibly with characters >255. It would be better to fix SciTE and leave
the meaning of the character notification unchanged as changing it will
break other applications. """

I absolutely agree with you.
I do not want to touch scintilla's internals.

Let me just mention one thing.
unicode point never works.
That means default Scite do not work with autocompletion on IME.
I think this is the kind of things only you can handle.

I want Korean IME on the spot on gtk.

Two AVIs attatched.

[Neil Hodgson]

Two AVIs attached.

Please don't attach videos to messages to the mailing list since they are very large, will not be viewed by most, and will take up room in their mail boxes. Place them on a video hosting service like YouTube and include a URL in the message.

Neil

[Neil Hodgson]

Post by johnsonj
Let me just mention one thing.
unicode point never works.
That means default Scite do not work with autocompletion on IME.

Did you try the change to SciTE I mentioned?

Neil

[johnsonj]

Sorry. I am a begginer.
Sorry I have troubled you.

I will try to change SciTE.

[johnsonj]

Is it right SCN_CHARADDED in SciteBase.cxx?
If so, It never trigger autocompletion.

[johnsonj]

it never triggers autocompletion on win32 too.
Even AddCharUTF(utfval, len, true) never does.

[Neil Hodgson]

Post by johnsonj
it never triggers autocompletion on win32 too.

Autocompletion is triggered in the edit pane by SCN_CHARADDED if the character is in the autocomplete.<lexer>.start.characters property or autocomplete.*.start.characters or if autocompleteword.automatic is set and the character is in the set of word characters. All of these may only work for ASCII.

In the output pane it is triggered by entering "$(".

With your videos I can't tell if you are manually triggering autocompletion with Complete Symbol or Complete Word or if it is automatically triggering while typing.

Neil

[johnsonj]

it is automatically triggering while typing.

My setting:
two dictionarys and
word.characters.$(file.patterns.text)=$(chars.alpha)$(chars.numeric)$(chars.accented)$(chars.hangul)

I set chars.hangul. it triggers autocompletion automatically while typing.
But it works only in TreatAsDBCS although somwhat tardy.
It never works in unicode.

If scintilla's autocompletion is desined not to work on IME chars.
My setting happens to work.

I wonder if it is right.

[Neil Hodgson]

Post by johnsonj
word.characters.$(file.patterns.text)=$(chars.alpha)$(chars.numeric)$(chars.accented)$(chars.hangul)
I set chars.hangul. it triggers autocompletion automatically while typing.

word.characters hasn't been designed to work for multi-byte characters since the "contains" method searches for a single byte.

Are chars.hangul defined in Code Page 949 or in UTF-8? It may only work if the encoding of chars.hangul matches the encoding of the document.

Neil

[johnsonj]

"In the output pane it is triggered by entering "$(“."

It is interesting. I have not known that.
I already know how to trigger autocompletion manually.
Even if it only works on ascii, that is no problem.
IME chars can not be typed into to trigger autocomplete manually.
As you see since IME chars needs composing.

[johnsonj]

""" Are chars.hangul defined in Code Page 949 or in UTF-8? It may only
work if the encoding of chars.hangul matches the encoding of the
document."""

Of course. I use it separately 949 or utf8.
How inconvenient!
I am going to discuss this problem later.

[johnsonj]

""'
word.characters hasn’t been designed to work for multi-byte characters
since the "contains” method searches for a single byte

"""

How astonishted! I cannot believe it.
I admit it as is.
But what is treatAsDBCS and Unicode conversion to notify?
Is it ready for IME?

[Neil Hodgson]

Post by Neil Hodgson
""'
word.characters hasn't been designed to work for multi-byte characters since the "contains" method searches for a single byte
"""
How astonishted! I cannot believe it.
I admit it as is.

Much of SciTE was developed by people that only use ASCII and some features only consider ASCII. Sometimes they work for other encodings but those cases may also fail.

Post by Neil Hodgson
But what is treatAsDBCS and Unicode conversion to notify?
Is it ready for IME?

The treatAsDBCS parameter was added by a Japanese user to support IMEs in this change set:
https://sourceforge.net/p/scintilla/code/ci/ce2d84311f4cae1f23420e8a4b0df7e758dcc5b6/

Neil

[johnsonj]

Thank you.
Now I see.
I guess maybe the japanese author intended for comparing aucomplete list.
But It proved not.

it is time to fix notify code.
Ignore treatAsDBCS.

that code is enough with following code.

NotifyChar((static_cast<unsigned char>(s[0]) << 8) |
static_cast<unsigned char>(s[1]));

More accurately

It should be replaced with follows
NotifyChar(static_cast<unsigned char>(s[0]));


this code is intended for triggering not for comparing.
Using the first byte of one char is enough to trigger autocompletion

I wonder what you think of.

[Neil Hodgson]

Post by johnsonj
NotifyChar((static_cast<unsigned char>(s[0]) << 8) |
static_cast<unsigned char>(s[1]));
More accurately
It should be replaced with follows
NotifyChar(static_cast<unsigned char>(s[0]));

NotifyChar is not just for autocompletion. It is a feature with already defined behaviour which clients rely on. This can not be changed without breaking other applications.

The way to fix autocompletion in SciTE is to change SciTE to understand multi-byte characters for autocompletion and other features like calltips.

Neil

[johnsonj]

Ok let it be as is.
It is good Scite will be enhanced.

But I insist to use notifyChar() in IME until when this situations will be
fixed;
It has no affect other codes.
It provide good user experience (How cool on the spot!)
If a problem happens, Let hangul users handle it.
it is ok just to delete the one line.

What do you think about?
Do you allow me to use notifyChar() in IME for the time being.

[Neil Hodgson]

But I insist to use notifyChar() in IME until when this situations will be fixed;
It has no affect other codes.
It provide good user experience (How cool on the spot!)
If a problem happens, Let hangul users handle it.
it is ok just to delete the one line.
What do you think about?
Do you allow me to use notifyChar() in IME for the time being.

Compatibility with existing applications is the top priority.

This is an incompatible change to the meaning of SCN_CHARADDED so I will not include it.

In ScintillaWin::HandleCompositionKoreanIME, the additional use of Editor::NotifyChar will be removed and the treatAsDBCS argument will be !IsUnicodeMode(). This should restore the meaning of SCN_CHARADDED on Windows.

Should the rest of the GTK+ IME code be used or will it be worse than the current situation without the NotifyChar?

Neil

[johnsonj]

Yes. I admit
I believe you will arrage that.

[Neil Hodgson]

The ScintillaWin changes have been committed.

There is a problem with ibus Korean on GTK+. This occurs before adding johnsonj's patch so it is not caused by the patch.

The problem occurs with ibus in Korean Hangul mode when the application is compiled with Clang with the address sanitizer turned on. A heap buffer overflow occurs inside IME code but it actually appears to be in the Thai IME. Possibly the Korean IME shares code with Thai. A failure log is attached to this mail. To reproduce build SciTE with address sanitiser, start it, choose Korean from ibus widget, type 'k' into SciTE. This failure may be a benign out-of-bounds memory issue which are common but it could also be the cause of further failure.

I am hesitant to add the Korean IME code from johnsonj because of this issue and the typing failure issue as it will be more difficult to debug keyboard problems if there is new code involved.

Neil

[johnsonj]

sorry for that.
I am scurunitizing for the codes.
I found my carelessness.

in PreeditChangedThis()

*pout = '\0';


I found I did not initialize *pout.
It may cause the stack overflow.

I will keep going on spotting another bug.
How difficult programming is!

Thank you very much for giving attention to KoreanIME.

[johnsonj]

I manged to change makefile and compiled scite as follows.
......................
clang++ -pthread -I/usr/include/gtk-2.0
-I/usr/lib/i386-linux-gnu/gtk-2.0/include -I/usr/include/atk-1.0
-I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0
-I/usr/include/gio-unix-2.0/ -I/usr/include/glib-2.0
-I/usr/lib/i386-linux-gnu/glib-2.0/include -I/usr/include/pixman-1
-I/usr/include/freetype2 -I/usr/include/libpng12 --std=c++0x -DNDEBUG -Os
-Wall -pedantic -DGTK -DSCI_LEXER -I ../include -I ../src -I ../lexlib
-c ScintillaGTK.cxx
.......................
I am not sure whether sanitize on or not.
I report It works with no problem on my XP.

My IME has CJK. no thai IME.

[Neil Hodgson]

Post by johnsonj
I manged to change makefile and compiled scite as follows.
......................
clang++ -pthread -I/usr/include/gtk-2.0 -I/usr/lib/i386-linux-gnu/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12 --std=c++0x -DNDEBUG -Os -Wall -pedantic -DGTK -DSCI_LEXER -I ../include -I ../src -I ../lexlib -c ScintillaGTK.cxx
.......................
I am not sure whether sanitize on or not.

The makefile includes the address sanitizer when building for debug:
make CLANG=1 DEBUG=1

Neil

[johnsonj]

àž«àž«àžŸàžàž”à¹€àž³àž«àžàž”à¹€àžžàž«à¹€à¹€
Wow ibus is very good. I like it
thai IME is good.
thai IME has something for korean IME to learn

I report no problem.

Maybe sanitizer off?

[johnsonj]

I will try it.
While my test I found one weird thing.
I happens to type in DBCS mode with unconcicousness.
Japnese IME gets My ibus gone.
localeVal[4] is not sufficient to Japanes IME.

I remember maxLenInpuIME[200]; this may be why.
Is there anyone who has used DBCS mode on IMEs?
I am not sure.

[johnsonj]

make clean.
I made it accoding your instruction.
result in a warning related with glib.h

Ecexpt DBCS mode, I see All works well yet.
I have one thing left to try.
I wll try latest mercurial version tomorrow.

[Neil Hodgson]

Post by johnsonj
make clean.
I made it accoding your instruction.
result in a warning related with glib.h

The warnings are just about glib using the "register" keyword which no longer does anything. Those warnings can be ignored and the executable should still be built OK.

Post by johnsonj
Ecexpt DBCS mode, I see All works well yet.
I have one thing left to try.
I wll try latest mercurial version tomorrow.

Address sanitizer will stop the application when an error occurs and write data out to standard error. Its easiest to see this by running SciTE from a command line.

For people using Scintilla in other applications, its a good idea to add an option for a build using address sanitizer since it finds many memory problems and slows the application down much less than alternatives like valgrind.

Neil