'elasticSearchRest': 'org.elasticsearch.client:elasticsearch-rest-high-level-client:7.9.3',

'elasticSearchTransport': 'org.elasticsearch.client:transport:7.9.3',

'gmaElasticsearchDao': "com.linkedin.datahub-gma:elasticsearch-dao-7:$gmaVersion",

'gmaElasticsearchIntegTest': "com.linkedin.datahub-gma:elasticsearch-dao-integ-testing-docker-7:$gmaVersion",

FROM python:3.8

COPY . .

RUN pip install --upgrade pip

RUN pip install elasticsearch

ENTRYPOINT ["python", "transfer.py"]

parser.add_argument('--disable-source-ssl', required=False, action='store_true', help='If set, disable source SSL.')

parser.add_argument('--disable-dest-ssl', required=False, action='store_true', help='If set, disable destination SSL.')

parser.add_argument('--create-only', required=False, action='store_true', help='If set, only create the index (with settings/mappings/aliases).')

ssl_context = create_ssl_context() if not args.disable_source_ssl or not args.disable_dest_ssl else None

source_ssl_context = ssl_context if not args.disable_source_ssl else None

dest_ssl_context = ssl_context if not args.disable_dest_ssl else None

image: elasticsearch:7.9.3

image: kibana:7.9.3

'.settings=$settings[0] | .mappings=$mappings[0]' > /tmp/data

curl -XPUT $ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT/$1 -H 'Content-Type: application/json' --data @/tmp/data

"docs/advanced/es-7-upgrade",

Checkout the list of breaking changes for [Elasticsearch 6](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/breaking-changes-6.0.html) and [Elasticsearch 7](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html). Following is the summary of changes that impact Datahub.

- Removal of mapping types (as mentioned [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html))

- Specify the maximum allowed difference between `min_gram` and `max_gram` for NGramTokenizer and NGramTokenFilter by adding property `max_ngram_diff` in index settings, particularly if the difference is greater than 1 (as mentioned [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html))

The following parameters are/were `optional` and hence automatically populated in the search query. Some tests that expect a certain search query to be sent to ES will change with the ES upgrade.

- `disable_coord` parameter of the `bool` and `common_terms` queries has been removed (as mentioned [here](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/breaking-changes-6.0.html))

- `auto_generate_synonyms_phrase_query` parameter in `match` query is added with a default value of `true` (as mentioned [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-match-query.html))

- In 7.9.3, Java High Level Rest Client instance needs a REST low-level client builder to be built. In 5.6.8, the same instance needs REST low-level client

- Document APIs such as the Index API, Delete API, etc no longer takes the doc `type` as an input

As mentioned in the docs, indices created in Elasticsearch 5.x are not readable by Elasticsearch 7.x. Running the upgraded elasticsearch container on the existing esdata volume will fail.

For local development, our recommendation is to run the `docker/nuke.sh` script to remove the existing esdata volume before starting up the containers. Note, all data will be lost.

To migrate without losing data, please refer to the python script and Dockerfile in `contrib/elasticsearch/es7-upgrade`. The script takes source and destination elasticsearch cluster URL and SSL configuration (if applicable) as input. It ports the mappings and settings for all indices in the source cluster to the destination cluster making the necessary changes stated above. Then it transfers all documents in the source cluster to the destination cluster.

You can run the script in a docker container as follows

We will create an "elasticsearch-5-legacy" branch with the version of master prior to the elasticsearch 7 upgrade. However, we will not be supporting this branch moving forward and all future development will be done using elasticsearch 7.9.3

RestClientBuilder restClientBuilder;

if (useSSL) {

restClientBuilder = loadRestHttpsClient(host, port, threadCount, connectionRequestTimeout, sslContext);

} else {

restClientBuilder = loadRestHttpClient(host, port, threadCount, connectionRequestTimeout);

return new RestHighLevelClient(restClientBuilder);

private static RestClientBuilder loadRestHttpClient(@Nonnull String host, int port, int threadCount,

int connectionRequestTimeout) {

.setHttpClientConfigCallback(httpAsyncClientBuilder -> httpAsyncClientBuilder.setDefaultIOReactorConfig(

IOReactorConfig.custom().setIoThreadCount(threadCount).build()));

setConnectionRequestTimeout(connectionRequestTimeout));

return builder;

private static RestClientBuilder loadRestHttpsClient(@Nonnull String host, int port, int threadCount,

int connectionRequestTimeout, @Nonnull SSLContext sslContext) {

.setHttpClientConfigCallback(httpAsyncClientBuilder -> httpAsyncClientBuilder.setSSLContext(sslContext)

.setSSLHostnameVerifier(new NoopHostnameVerifier())

.setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(threadCount).build()));

setConnectionRequestTimeout(connectionRequestTimeout));

return builder;

"max_ngram_diff": 17,

"max_ngram_diff": 17,