Elasticsearch Indices keeps getting lost

Question

I have an Elasticsearch cluster on AWS ec2 instance. It is t3.small with 2 vcores and 2 GB ram. I have installed Elasticsearch & kibana. For extensions, I have installed Heartbeat and Metricbeat. The database I'm working with is mongo DB and all my data is no-SQL. I feed my engine from my MongoDB cluster which is in my local machine with a script. I feed my engine and run the queries from my app and also from the console. So far so good. Everything is fine. Well, the cluster is always yellow it is not green.
The problem starts after hitting multiple requests on the engine. After 50 or 60 search queries the data just disappears. Well somehow my engine is forcefully dumping my indices and it's not being able to restore those data ( obviously I have no snapshot and no restore point ) and I keep getting lose those data. I have to manually feed the engine again and again. Well at first I had 1 GB ram so I thought upgrading would fix the issue but after upgrading to 2 GB ram it didn't stop. Well, now the data stays there for some more time.
So here are my DB configs.

• I have 70K + no SQL documents.
• Which contains text and geo_point types
• I make post request on my engine through my front end application.
• I don't have logstash installed, but metricbeat is not showing any error logs.
• All my elastic search engine setup is for Testing purposes this is not the production mode.
• We will upgrade when we go to the production mode.

So I need to know

• What is the reason behind this and
• how to prevent this huge data loss

So please help me or just suggest how to solve this huge problem.
Thank you

Answer 1

Elastic documentation leaves much to be desired, but if, by "engine" you are referring to the engines that show up in the Enterprise Search/App Search tab (Open a deployment from the Elastic main dashboard, navigate to Overview link under the Search section, and from there to the App Search link under the Enterprise Search section), then this might help.

When an Engine is created, if you select the search engine type as being Elasticsearch index-based, and you select an index that does not start with search-, then it will use your selected index, but create an alias with the name search-yourindexname-alias. If yourindexname gets reindexed, that alias may disappear, and the engine will look like there are no records there. The solution should be to re-create the disappeared alias on yourindexname.

Answer 2

Ideally first thing you should do is to make cluster green.

To see the exact elasticsearch error that is causing this situation, you should look at elasticsearch.log file. It will contain exact error causing it.

One way to keep cluster data safe is to take regular snapshots and restore incase of data loss. Details of snapshots procedure can be found here.