#2497 Rectangular selection indent incorrect selection modes and positions

[Michael Heath]

• Windows 11
• ScITE: 5.5.8 Scintilla: 5.5.8 Lexilla: 5.4.6

The issue is based on:

BUG Incorrect tab behavior in column mode

caused by a change-set committed at

Remove unnecessary InvalidateStyleRedraw() for Message::SetUseTabs

Symptoms:

• Carets can move to incorrect positions.
• Undo changes selection mode so Redo is different.

---

Note: Preview of code converts tabs to spaces. I expect the results to remain the same.

1. Test start of rectangle at column 0:

-
    -

1. Rectangle select downwards and tab to indent result

    |-
    |   -

The pipe character represent the carets.
The change-set sets a thin selection.

1. Reset to step 1. Rectangle select upwards and tab to indent result.

    -|
        |-

The 1st caret is 1 position too far.

Another symptom from the change-set is undo reverts the selection mode to stream mode as the indenting was done with stream mode. So undo and redo does this with the select downwards continuing with step 2.

    |-
        |-

The undo seems to forget the rectangle mode and remembers the stream mode so the result is different.

I have attached possible fix. I tested the IDM_HELP code in SciTE and it seems to pass OK.

[Neil Hodgson]

The patch appears to me to change behaviour for too many situations (any call to SetRectangularRange) to be safe.

A change restricted to Editor::Indent is less likely to cause new problems.

A branch inside Editor::Indent that specializes the rectangular case explicitly should allow tighter control. Operating on a copy of the initial selection could avoid interference from rectangular selection recalculations. This would likely duplicate the lengthy calculations for each selection although the specific issue (tab key) is just for Indent(true, false) which is a shorter subset of the code.

[Neil Hodgson]

The example shows the effect of trying to maintain selection rectangularity over the indent operation.

The current indent code is mostly designed to handle stream selection. It chooses the end of indentation as the best place to position the caret afterwards as it simplifies some common actions like adding spaces for fine control over indentation or adding to the line text. This choice is more advantageous when there are multiple selections.

In the example, the lines initially get empty selections after indentation: after one tab for line 1 and after two tabs for line 2. Then these end points are converted to a thin rectangular selection. Calculating rectangular ranges is asymmetric to match the direction of dragging when performed by mouse. For a thin selection, the anchor is chosen so a forward selection chooses the column after 1 tab and a reverse selection chooses the column after 2 tabs, leading to selecting after the '-' on line 1.

The purpose of the SelTypes::thin mode is to produce selections that contain no text so are suitable for following operations, The patch presented here may create non-empty selections so is not really upholding the SelTypes::thin mode.

As mentioned on the earlier issue, the best solution here is probably a separate rectangular code path that achieves more rectangular results, similar to the multi-line path.

A smaller change would be to first abandon rectangular mode, converting the selection to multiple stream selections before indenting each of these.

[Michael Heath]

https://github.com/notepad-plus-plus/notepad-plus-plus/issues/17526 appears to be related to this issue.