Discussion:
Decorations
(too old to reply)
Neil Hodgson
2007-04-01 23:32:07 UTC
Permalink
Projects based on Scintilla have been using indicators to mark
areas of text that contain errors, search matches, URL hotspots,
misspellings, etc. These areas are discovered through some user
initiated command or by running a background process rather than as
part of lexing. Such marks are often fairly sparse: a HTML syntax
checker may show 5 errors in a whole file. Its a little complex using
indicators in this way as the lexing state has to be preserved while
setting the indicator and there is only 1 to 3 bits available for this
state.

I'm thinking of adding the capability to have a set of indicators
that are not stored in the styling bytes but rather in a compact
format similar to run length encoding. They may be called "Extended
Indicators" or "Decorations". The implementation would be based on
SinkWorld's RunStyles class. The existing indicator drawing code would
be reused with the same styles available for each type.

A decoration would be reasonably stable: inserting or deleting text
outside a decoration leaves it alone; insertions inside a decoration
show the decoration, and insertions at the edges don't show the
decoration. Decorations would not be remembered in the undo history.
Since they are not lexer based there wouldn't be any 'decorations
valid' range remembered by Scintilla nor an equivalent to
SCN_STYLENEEDED. Containers would manage this themselves through
SCN_MODIFIED.

Decorations could either be implemented completely in the view
layer (simpler but only visible in one view) or with the locations
stored in the document and visual appearance in the view. There would
be a defined drawing order and some could be drawn underneath the text
and others over the text. The number of decoration types allowed could
be open ended or fixed. If fixed at 32 it would be simpler to treat as
a bit set for notifications and calls. Mouse clicks in decorations
could be notified to the application. There would be APIs to set a
decoration over a range, to clear a decoration over a range, see if a
decoration or any decorations are at a position and to find the start
and end of a decoration range.

Neil
Armel Asselin
2007-04-02 17:17:49 UTC
Permalink
Post by Neil Hodgson
Projects based on Scintilla have been using indicators to mark
areas of text that contain errors, search matches, URL hotspots,
misspellings, etc. These areas are discovered through some user
initiated command or by running a background process rather than as
part of lexing. Such marks are often fairly sparse: a HTML syntax
checker may show 5 errors in a whole file. Its a little complex using
indicators in this way as the lexing state has to be preserved while
setting the indicator and there is only 1 to 3 bits available for this
state.
I'm thinking of adding the capability to have a set of indicators
that are not stored in the styling bytes but rather in a compact
format similar to run length encoding. They may be called "Extended
Indicators" or "Decorations". The implementation would be based on
SinkWorld's RunStyles class. The existing indicator drawing code would
be reused with the same styles available for each type.
A decoration would be reasonably stable: inserting or deleting text
outside a decoration leaves it alone; insertions inside a decoration
show the decoration, and insertions at the edges don't show the
decoration. Decorations would not be remembered in the undo history.
Since they are not lexer based there wouldn't be any 'decorations
valid' range remembered by Scintilla nor an equivalent to
SCN_STYLENEEDED. Containers would manage this themselves through
SCN_MODIFIED.
Decorations could either be implemented completely in the view
layer (simpler but only visible in one view) or with the locations
stored in the document and visual appearance in the view. There would
be a defined drawing order and some could be drawn underneath the text
and others over the text. The number of decoration types allowed could
be open ended or fixed. If fixed at 32 it would be simpler to treat as
a bit set for notifications and calls. Mouse clicks in decorations
could be notified to the application. There would be APIs to set a
decoration over a range, to clear a decoration over a range, see if a
decoration or any decorations are at a position and to find the start
and end of a decoration range.
very interesting...
with run-length encoding having fixed 32 bits fields would probably good and
simple.
I would rather put them in document as it's done currently.

Armel
Josiah Carlson
2007-04-02 18:08:12 UTC
Permalink
Post by Neil Hodgson
Projects based on Scintilla have been using indicators to mark
areas of text that contain errors, search matches, URL hotspots,
misspellings, etc. These areas are discovered through some user
initiated command or by running a background process rather than as
part of lexing. Such marks are often fairly sparse: a HTML syntax
checker may show 5 errors in a whole file. Its a little complex using
indicators in this way as the lexing state has to be preserved while
setting the indicator and there is only 1 to 3 bits available for this
state.
[snip]

Sounds great to me. Allowing so many indicators would allow for syntax
highlighting while also supporting SubEthaEdit-like pair/group
programming (something I've been looking to add to my scintilla-based
editor for quite a while), along with different colors for console
stdout/stderr/expected input in shells (also something I've been looking
to add for a while).

Thank you for considering this.


- Josiah
Shane Caraveo
2007-04-02 22:31:44 UTC
Permalink
Post by Josiah Carlson
Post by Neil Hodgson
Projects based on Scintilla have been using indicators to mark
areas of text that contain errors, search matches, URL hotspots,
misspellings, etc. These areas are discovered through some user
initiated command or by running a background process rather than as
part of lexing. Such marks are often fairly sparse: a HTML syntax
checker may show 5 errors in a whole file. Its a little complex using
indicators in this way as the lexing state has to be preserved while
setting the indicator and there is only 1 to 3 bits available for this
state.
[snip]
Sounds great to me. Allowing so many indicators would allow for syntax
highlighting while also supporting SubEthaEdit-like pair/group
programming (something I've been looking to add to my scintilla-based
editor for quite a while), along with different colors for console
stdout/stderr/expected input in shells (also something I've been looking
to add for a while).
That would be great, we have some hacky patches to get coloring for
stdout/err/in that do not always work well.

Shane
Simon Steele
2007-04-02 21:30:44 UTC
Permalink
Neil,
Post by Neil Hodgson
I'm thinking of adding the capability to have a set of indicators
that are not stored in the styling bytes but rather in a compact format
similar to run length encoding. They may be called "Extended Indicators"
or "Decorations". The implementation would be based on SinkWorld's
RunStyles class. The existing indicator drawing code would
be reused with the same styles available for each type.
Sounds excellent, the current system can be complicated to use and very
restrictive when using complicated lexers like the hypertext one.

As far as storage, I can't see a view with >32 indicators as anything
other than disastrous from a usability point of view so surely a 32-bit
field would be adequate?

I'm presuming lexers will be able to set decorations as well as making
them available via an API. In this case would it not be better to store
the state with the document, I don't think we currently lex multiple times
for multiple views?

Simon.
Neil Hodgson
2007-04-03 00:22:53 UTC
Permalink
Post by Simon Steele
As far as storage, I can't see a view with >32 indicators as anything
other than disastrous from a usability point of view so surely a 32-bit
field would be adequate?
I was thinking that it could be allocated for different purposes
since you don't want to have a conflict between lexers and containers.
The first 8 bits could be for lexers. This includes the existing 3
bits currently available to lexers. For a transition period,
indicators 0, 1, and 2 are still stored in the style bytes. but will
be treated as equivalent by the new API. Lexers are then updated to
use bits 3 to 7 and after the transition, bits 0,1,and 2 use
RunStyles. If anyone is using indicators for dense styles that are not
suitable for RunStyles (such as marking the start of every lexeme)
then they should comment.

The order of drawing may be important for some decorations - such
as highlighting the spelling mistake under the cursor while retaining
the original mistake decorations. There is also the possibility of a
multi-valued decoration: maybe you have 5 levels of issue severity.
While this can be represented as 5 separate decorations, it may be
better to treat it as a simple decoration with 6 values (including 0
as none).
Post by Simon Steele
I'm presuming lexers will be able to set decorations as well as making
them available via an API. In this case would it not be better to store
the state with the document, I don't think we currently lex multiple times
for multiple views?
Yes, lexing is handled by the document. Looks like decorations have
to be stored in the document.

Neil
Eric Promislow
2007-04-03 00:04:00 UTC
Permalink
Sounds fine for our needs. We'd prefer to have the decorations
associated with the document -- it's easier for apps that
can provide more than one concurrent view to a document, and
it then pushes the responsibility for managing multiple views
to each app, which makes sense to me.

The SCN_MODIFIED event works if getting the list of decorations
at a range (which might be one-byte long) would be efficient.

- Eric
Post by Neil Hodgson
Projects based on Scintilla have been using indicators to mark
areas of text that contain errors, search matches, URL hotspots,
misspellings, etc. These areas are discovered through some user
initiated command or by running a background process rather than as
part of lexing. Such marks are often fairly sparse: a HTML syntax
checker may show 5 errors in a whole file. Its a little complex using
indicators in this way as the lexing state has to be preserved while
setting the indicator and there is only 1 to 3 bits available for this
state.
I'm thinking of adding the capability to have a set of indicators
that are not stored in the styling bytes but rather in a compact
format similar to run length encoding. They may be called "Extended
Indicators" or "Decorations". The implementation would be based on
SinkWorld's RunStyles class. The existing indicator drawing code would
be reused with the same styles available for each type.
A decoration would be reasonably stable: inserting or deleting text
outside a decoration leaves it alone; insertions inside a decoration
show the decoration, and insertions at the edges don't show the
decoration. Decorations would not be remembered in the undo history.
Since they are not lexer based there wouldn't be any 'decorations
valid' range remembered by Scintilla nor an equivalent to
SCN_STYLENEEDED. Containers would manage this themselves through
SCN_MODIFIED.
Decorations could either be implemented completely in the view
layer (simpler but only visible in one view) or with the locations
stored in the document and visual appearance in the view. There would
be a defined drawing order and some could be drawn underneath the text
and others over the text. The number of decoration types allowed could
be open ended or fixed. If fixed at 32 it would be simpler to treat as
a bit set for notifications and calls. Mouse clicks in decorations
could be notified to the application. There would be APIs to set a
decoration over a range, to clear a decoration over a range, see if a
decoration or any decorations are at a position and to find the start
and end of a decoration range.
Neil
_______________________________________________
Scintilla-interest mailing list
http://mailman.lyra.org/mailman/listinfo/scintilla-interest
Neil Hodgson
2007-04-03 02:40:04 UTC
Permalink
Post by Eric Promislow
The SCN_MODIFIED event works if getting the list of decorations
at a range (which might be one-byte long) would be efficient.
Retrieving the value at a particular position requires a binary
search of the run starts sequence. If a separate RunStyles is used for
each type of decoration then this must be repeated for each type.
Since each run has a 32 bit integer value, all of the decorations
could be encoded into a single RunStyles but this is harder to process
in other ways such as finding the end of a particular type and becomes
inefficient when multiple denser decorations are combined. Determining
the set of bits that are set for some portion of a range could be made
reasonably efficient although I hadn't thought of that as an API.

Neil
Neil Hodgson
2007-04-04 01:19:06 UTC
Permalink
There is an initial implementation available from
http://www.scintilla.org/dscite.zip

The make and project files haven't been updated yet. The new source
files are in scintilla/src and are RunStyles.cxx and Decoration.cxx.
The Decoration code may be merged into Indicator. The maximum
indicator (INDIC_MAX) has been increased from 7 to 31 with indicators
0 to 7 still using bits in the style bytes and indicators 8 to 31
using the new decoration code. Indicators above 15 are currently drawn
under the text but this will probably become an attribute of each
indicator. The API is

# Set the indicator and value used for IndicatorFillRange and
IndicatorClearRange
fun void IndicatorSetCurrent=2500(int indicator, int value)

# Turn a indicator on over a range.
fun void IndicatorFillRange=2501(int position, int fillLength)

# Turn a indicator off over a range.
fun void IndicatorClearRange=2502(int position, int clearLength)

# Are any indicators present at position?
fun int IndicatorAllOnFor=2503(int position,)

# What value does a particular indicator have at at a position?
fun int IndicatorValueAt=2504(int indicator, int position)

# Where does a particular indicator start?
fun int IndicatorStart=2505(int indicator, int position)

# Where does a particular indicator end?
fun int IndicatorEnd=2506(int indicator, int position)

Because of the two parameter limit for Scintilla APIs,
IndicatorFillRange and IndicatorClearRange implicitly affect the
indicator set in IndicatorSetCurrent. The 'value' parameter to
IndicatorSetCurrent is to allow multi-valued indicators in the future
and should always be set to 1 for now.

None of the new calls can affect the existing indicators stored in
style bytes. While the implementation could try to hide the
differences, it is more work and doesn't appear that useful if style
byte indicators are going to go away.

There are no creation or deletion methods: decorations appear when
a range is filled and disappear when there are no filled ranges.

Neil
Neil Hodgson
2007-04-07 03:11:02 UTC
Permalink
There is an update for the decoration code available from CVS and from
http://scintilla.sourceforge.net/scite.zip Source

Changing indicators now flow through Document to all views and
Accessor gains an IndicatorFill method so that lexers can set
indicators. While I have been editing LexPython to suite, it hasn't
been checked in since it is still uncertain how to do this best. One
problem is that the previous implementation minimized the number of
SC_MOD_CHANGESTYLE notifications by buffering these into 4000 byte
blocks but the indicator calls occur for each change and report the
whole range asked for rather than the range actually changed. There is
one optimization where the whole range asked for is already in the
requested style but that should be optimized further. There may be a
new SC_MOD_CHANGEINDICATOR to allow each view to redisplay and the
container to see these if wanted. Since changing indicators does not
change layout, SC_MOD_CHANGEINDICATOR may be a lighter weight event.

All the build files should now include Decoration and RunStyles.

Neil
Neil Hodgson
2007-04-19 05:41:53 UTC
Permalink
Some optimizations implemented for decorators. When a call is made
to change a range to a value it already has then no modifications are
made, the display is not repainted and the container does not receive
a notification. When the range is already partly in the value then the
start and end parts of the range that already have this value are
trimmed off for updates.

A new modification flag SC_MOD_CHANGEINDICATOR is used when an
indicator is changed instead of SC_MOD_CHANGESTYLE. This allows
containers to ignore these by changing the event mask. Since adding
and removing indicators does not affect text layout, these are faster.

No lexers use the new indicator code yet but I have been
experimenting with the Python lexer. It currently resets the whole
range being lexed and then fills in the bad indentation ranges. This
is not optimal as there will often be no need to change the bad
indentation indicators so this will probably change to keeping a
current state flag while lexing and filling whenever the state
changes. This is still not as efficient as the current code which
buffers up indicator changes with style changes but will probably be
OK.

Committed to CVS.

Neil
Neil Hodgson
2007-04-23 03:05:24 UTC
Permalink
I have committed an updated Python lexer that uses the new
indicator API. The documentation is updated to deprecate style byte
indicators in favour of 'modern' indicators. Each indicator may be set
to draw under the text or over (default) with SCI_INDICSETUNDER.

If this implementation of indicators is not strong enough for some
uses then this should be discussed now while there is little code that
will be broken if indicators are changed. Particularly useful would be
seeing if it helps implement a feature like snippets where there may
be multiple placeholders of various types in the text.

The Scriptol lexer also sets indicators so this should be changed
by someone that uses Scriptol. If you know of any other lexers that
set indicators then reply so they can be fixed or dropped.
LexPython.cxx can be used as an example to change lexers that are not
distributed with Scintilla. While style byte indicators will work in
the next release, they will stop working in the future so that all 8
bits in a style byte can be used for lexical states.

Available from CVS and from

http://scintilla.sourceforge.net/scite.zip Source
http://scintilla.sourceforge.net/wscite.zip Windows executable

Neil

Continue reading on narkive:
Loading...