The 30 second answer
Less than you might think. Elastic's guidance is to set the minimum and maximum heap equal, at no more than half the node's memory, and below the compressed object pointer threshold, which is about 26 gigabytes on most systems. Lucene reads its immutable segment files through memory mapping, so the rest of the memory serves as the operating system's file cache, which is what keeps searches fast.
The guidance. Elasticsearch's JVM settings page says the default sizing suits most production clusters. To override it, set Xms and Xmx to the same value, no more than 50% of the memory available to the node, and below the threshold for compressed ordinary object pointers: "26GB is safe on most systems and can be as large as 30GB on some." The startup log reports whether compressed pointers are in use.
Why not give it all to the heap. Elasticsearch also uses memory outside the heap, for example network buffers, and the search-speed guide says it "relies heavily on the filesystem cache", recommending that at least half the available memory goes to it so hot parts of the index stay in RAM. Because segments are immutable, cached file pages stay valid until a merge replaces the segment.
Memory mapping. Elasticsearch stores indices with a memory-mapped directory by default, so Linux must allow enough mappings: the virtual memory setting vm.max_map_count has to be raised above the operating system default, as the install docs describe.
JVM view. A larger heap means more live data for the collector to trace and potentially longer pauses, and above the compressed-pointer threshold every reference grows from 4 to 8 bytes, so a bigger heap can hold fewer objects. Watch heap after GC, GC pause times, and the circuit breakers that reject requests which would use too much memory.
What fills the heap. Field data on text fields, which should be avoided, large aggregations, big bulk requests, and caches. High-cardinality terms aggregations, as in Zalando's incident, are both CPU-heavy and memory-heavy.
The general JVM side is on What are the JVM memory areas? and How do G1 and ZGC compare?.