Sqlite Full-text Search Unicode In Android

I am creating a table in SQLite using fts(3 or 4) CREATE VIRTUAL TABLE Demo1 USING fts3(content TEXT); insert into Demo1 values('Hồ Thanh Long'),('Nguyễn Văn A') When search

Solution 1:

You must create the FTS table with a tokenizer that can handle Unicode characters, i.e., ICU or UNICODE61.

Please note that these tokenizers might not be available on all Android versions, and that the Android API does not expose any functions for adding user-defined tokenizers.

Solution 2:

The default "simple" tokenizer for android supports unicode:

where eligible characters are all alphanumeric characters and all characters with Unicode codepoint values greater than or equal to 128.

It just doesn't do anything else. I'm not sure even the Unicode tokenizers would do the mapping you require. (i.e. recognize 'Hồ' as both 'Hồ' and 'Ho' when queried.)

Indeed, the demo recognized 'Hồ' when you queried it; it just didn't return it when you queried 'Ho' because it didn't recognize them as equivalents. If you are working with a limited set of supported Unicode characters, you could implement your own mapping, and save the "plain ASCII text" in a separate column to search on separately.