//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//

==== Data model and ElasticSearch 7

Since Apache Unomi version 1.5.0 we decided to upgrade the supported ElasticSearch version to the 7.4.2.

To be able to do so, we had to rework the way the data was stored inside ElasticSearch.

Previously every items was stored inside the same ElasticSearch index but this is not allowed anymore in recent ElasticSearch versions.

Since Apache Unomi version 1.5.0 every type of items (see section: link:#_items[Items]) is now stored in a dedicated separated index.


==== API changes

To be able to handle the multiple indices the Persistence API implementation
(https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java[ElasticSearchPersistenceServiceImpl])
have been adapted and simplified.

The good news is that there is no API changes, the persistence API interface didn't changed.

Any custom Apache Unomi plugins or extensions should continue to work on Apache Unomi 1.5.0.

The only notable changes are located at the
https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java[ElasticSearchPersistenceServiceImpl Java class].
This class should not be use directly, instead you should use OSGI service dependency injection using the interface https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-spi/src/main/java/org/apache/unomi/persistence/spi/PersistenceService.java[PersistenceService].

But if you are interested in the implementation changes:

. The property `index.name` have been renamed to `index.prefix`.
Previously used for the single one index name, now every index is prefixed using this property. (`context-` by default)
. We removed the property `index.names` originally used to create additional indices (used by the geonames DB for exemple).
This property is not needed anymore because the index is automatically created by the peristence service when the mapping configuration is loaded.
Example of mapping configuration file: (https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/extensions/geonames/services/src/main/resources/META-INF/cxs/mappings/geonameEntry.json[geoname index mapping])

Because of this changes the geonames DB index name is now respecting the index naming with prefix like any other item type.
Previously named: `geonames` is now using the index name `context-geonameentry`
(see: link:#_installing_geonames_database[Documentation about geonames extension]).

==== Migration steps

In order to migrate the data from ElasticSearch 5 to 7, Unomi provides a migration tool that is directly integrated.

In this migration the following is assumed:

- the ElasticSearch 5 cluster installation is referred to as the `source`
- the ElasticSearch 7 cluster installation is referred to as the `target`
- the Unomi 1.4 cluster installation is completely stopped
- the Unomi 1.5 cluster installation has never been started (just uncompressed)
- the Unomi 1.5 cluster installation has been configured to connect to the `target` (ElasticSearch 7) cluster

It is HIGHLY RECOMMENDED to perform a full cluster backup/snapshot of the `source` clusters (including ElasticSearch and
Unomi clusters), and ideally to perform the migration on a restored snapshot of the `source` cluster. For more information
on ElasticSearch 5 snapshots and restore you can find it here:

    https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-snapshots.html

The way the migration works is that both ElasticSearch 5 AND an ElasticSearch 7 clusters (or just single nodes) will
be started at the same time, and data will be migrated from the ES 5 to the ES 7 cluster. Note that it is possible to use
a single node for both the `source` and the `target` clusters to - for example - perform the migration on a single
machine. If you choose to do that you will have to adjust port numbers on either the `source` or `target` cluster node.
Changing ports requires a restart of the ES cluster you are modifying. In this example we will illustrate how to migrate
by modifying the `source` cluster node ports.

So in the `source` 's ElasticSearch 5 `config/elasticsearch.yml` file we have modified the default ports to:

    transport.tcp.port: 9310
    http.port: 9210

Make SURE you change the ports out of the default 9200-9205 and 9300-9305 range (or whatever your cluster uses) otherwise
both clusters will attempt to merge!

On the `target` ElasticSearch 7 cluster configuration you will need to add the following setting in the `config/elasticsearch.yml`:

    reindex.remote.whitelist: "localhost:9210"

Replace "localhost:9210" which whatever location your `source` cluster is available at. Restart or start your
`target` ElasticSearch 7 cluster.

Important: Make sure you haven't started Apache Unomi before (using the `unomi:start` command or the autostart command
line parameter) otherwise you will need to restart your Apache Unomi installation from scratch. The best way to be sure
of that is to start a new Unomi install by uncompressing the archive and not launching it.

You can then start both instances of ElasticSearch 5 and ElasticSearch 7 and finally start Apache Unomi using:

    ./karaf

Once in the console launch the migration using the following command:

    migrate 1.4.0

Note: the 1.4.0 version is the starting version. If you are starting from a different version (for example a fork), make
sure that you know what official version of Apache Unomi it corresponds to and you can use the official version number
as a start version for the migration.

Follow the instructions and answer the prompts. If you used the above configuration as an example you can simply use the
default values.

Be careful because the first address that the tool will ask for is the `target` (ElasticSearch 7) cluster, not the
ES 5 one.

Note that it is also possible to change the index prefix to be different from the default `context` value
so that you could host multiple Apache Unomi instances on the same ElasticSearch cluster.

Important note: only the data that Apache Unomi manages will be migrated. If you have any other data (for example Kibana
or ElasticSearch monitoring indices) they will not be migrated by this migration tool.

Once the migration has completed, you can start the new Unomi instance using:

    unomi:start

You should then validate that all the data has been properly migrated. For example you could issue a command to list
the profiles:

    profile-list
