How the ElasticSearch works when the data is encrypted at rest

Question

I have my data in DynamoDB. To enable full-text search, I'm using a Lambda function to forward that to ES index. But I would like to keep the data encrypted if possible without compromising my search efficiency. But I'm not sure, how my search will work here. I couldn't find any documents / articles how the search works internally when the data is encrypted at rest.

Say For example:

{"message": "Hello World"} 

When the data is encrypted, It will be something like this "asdasdeiurnjiluoikpourqjrqurioqurioimlkhoih"

When I send a Get Request to search for "Hello World". How this will respond.

Any help / directions are really appreciated. Thanks.

Answer 1

There is no easy answer. If you want your data to be properly encrypted and properly protected, any of the possible solutions would weaken the level of security. To start with, think better of use cases:

1. What are the attributes you need to allow for full-text search? Maybe it'd be possible to separate them so they won't form personally identifiable information and you'd be able to make a full-text search by them. That'd allow keeping such things in Elasticsearch (or just any RDB that has full-text indexes).
2. What are typical search queries you expect that'd lead to using a full-text index? Maybe you can narrow it down to using chunks of your data for a precise match (e.g. searching by "john" and finding "John Smith" as it may have two chunks: "john" and "smith") -- these chunks, in turn, could be properly encrypted so you need to find just one of them for a given entry.
3. What are the precise data protection requirements according to the risks analysis for your system? Maybe you are allowed to use in-application encryption and have an in-memory index of plain-text data while using encryption to persist data-at-rest.

Here are questions related to your topic that could be also helpful:

• https://dba.stackexchange.com/questions/23908/how-to-search-a-mysql-database-with-encrypted-fields
• https://medium.com/@ZeroDB_/scalable-full-text-search-over-encrypted-data-cb2b5dd5bce2
• Full text search on encrypted data
• Encrypted Fields & Full Text Search, Best Approach?
• Is it possible to use full text search on encrypted column in SQL Server 2008