Java Object Cache: Picking One and Not Getting Burned by It

A Java object cache keeps computed or fetched objects in memory so you do not pay for them twice. The decision that matters is not which library — it is whether the cache lives inside your JVM or outside it, because that choice determines what happens when you run more than one instance.

Get that wrong and you get a class of bug that is nearly impossible to reproduce: correct behaviour on your machine, inconsistent behaviour in production, depending entirely on which instance served the request.

Table of contents

• In-process against distributed
• What to use in-process
• Sizing and eviction
• Measuring whether it helps
• The invalidation problem
• The memory consequences
• How this fits the rest of the stack
• FAQ

In-process against distributed

In-process caches live in the JVM heap. Lookups are a hash map access — nanoseconds, no serialisation, no network. Each instance has its own copy, and the cache dies when the process restarts.

Distributed caches live in a separate service. Every instance sees the same data, entries survive restarts, and every lookup costs a network round trip plus serialisation on both ends.

The performance gap is enormous — roughly nanoseconds against roughly a millisecond. That gap is why in-process should be your default and distributed should be a decision you can justify.

The thing that forces distributed is not performance, it is correctness. If a cached value must be consistent across instances, or invalidation must take effect everywhere immediately, an in-process cache cannot do it. That is the whole test.

What to use in-process

The realistic options, and the guidance is unusually clear here:

• Caffeine — the current default for new code. High hit rates from its admission policy, good concurrency, and a clean API. This is what to reach for unless something else is imposed.
• Guava Cache — Caffeine’s predecessor, still everywhere in existing codebases. Fine, and Caffeine outperforms it on both throughput and hit rate. Not worth migrating for its own sake; worth choosing Caffeine for new code.
• ConcurrentHashMap — not a cache. No eviction, no expiry, no size bound. Perfectly good for a small fixed set of values loaded at startup, and a memory leak for anything keyed on user input.
• Ehcache and JCS — older, heavier, with disk persistence and distribution options. Choose deliberately, not by default.

That third point is where most accidental damage happens. A ConcurrentHashMap keyed on something unbounded — a user id, a search query, a request path — grows without limit until the heap is exhausted. It looks like a cache and has none of a cache’s safety properties.

Sizing and eviction

Every cache needs a bound. The question is which kind.

Cache<String, User> cache = Caffeine.newBuilder()
    .maximumSize(10_000)
    .expireAfterWrite(Duration.ofMinutes(10))
    .recordStats()
    .build();

Three settings, each doing distinct work:

• maximumSize bounds the entry count. Simple, and it assumes entries are roughly the same size. Use maximumWeight with a weigher when they are not — caching HTML fragments that range from 1KB to 5MB by count is how you exhaust a heap while staying under your limit.
• expireAfterWrite bounds staleness. Every entry is discarded a fixed time after it was written, regardless of use.
• expireAfterAccess discards entries that have not been read recently. Different meaning — it bounds idleness, not staleness. An entry read constantly never expires, which is wrong if the underlying data changes.

Prefer expireAfterWrite when the source data can change. expireAfterAccess is for pure computations where the value cannot become wrong.

recordStats is worth enabling from the start. A cache you cannot measure is a cache you cannot tune, and the hit rate is the only number that tells you whether it is doing anything.

Measuring whether it helps

Caches are added on the assumption they help and are rarely checked. Expose the statistics:

CacheStats stats = cache.stats();
log.info("hit rate {} evictions {} avg load penalty {}ms",
    stats.hitRate(), stats.evictionCount(),
    stats.averageLoadPenalty() / 1_000_000);

What the numbers mean in practice:

• Hit rate below about 50% — the cache is mostly overhead. Either the key space is too large for the size limit, or entries expire before they are reused.
• High eviction count with a low hit rate — the cache is too small and is thrashing. Raising the size may help; a key space that is genuinely unbounded will not be fixed by any size.
• A high average load penalty — each miss is expensive, which means the cache is valuable when it hits and worth investing in.

A cache with a 20% hit rate on cheap lookups is negative value: it costs memory, adds a code path, and creates staleness for almost no saving. Removing caches is a legitimate optimisation and it happens far less often than it should.

The invalidation problem

The genuinely hard part. Three strategies, in increasing order of correctness and complexity:

1. Time-based expiry only. Accept that data is stale for up to the TTL. Simple, predictable, and correct for anything where a few minutes of staleness is acceptable — which is more things than people assume.
2. Explicit invalidation on write. When you update the underlying record, evict the key. Correct within one instance. Across several instances, every other instance still holds the old value until its TTL expires.
3. Distributed invalidation. Publish an eviction message that every instance consumes. Correct across instances, and now you own a message channel and its failure modes.

Most systems should use the first, and reach for the second where a stale read is visibly wrong to a user. The third is a real commitment and should follow evidence that the second is insufficient.

The trap in the second strategy is thinking it works when you have more than one instance. It does not, and it looks like it does in every test that runs a single process. Behaviour depending on which instance handled the request is the signature of exactly this bug.

The memory consequences

A cache is a deliberate memory leak with a bound. Getting the bound wrong has consequences beyond running out of heap.

• Long-lived cached objects are promoted to the old generation, so they are collected only by major collections. A large cache means more major GC work and longer pauses.
• Cached objects hold references to everything they point at. Caching an entity that lazily references a large graph can retain far more than the entry appears to weigh.
• Soft references are not a solution. Letting the collector evict entries under pressure sounds elegant and produces unpredictable pauses and hit rates. Use an explicit size bound.

The practical guidance: bound the cache explicitly, size it relative to your heap rather than to a round number, and look at GC behaviour after adding a large one. A cache that improves average latency while adding a 400ms pause every few minutes may not be a win.

And size it against the container’s memory limit rather than the machine’s. A JVM in a container with a 1GB limit and a heap sized for the host is how services get killed by the runtime with no Java-level error at all.

How this fits the rest of the stack

The recurring shape here is that a cache is easy to add and its failure modes only appear at scale — inconsistency across instances, GC pauses under memory pressure, staleness nobody planned for. All three are much easier to catch when the memory limit your process actually runs against is a number you chose rather than one you inherited. Java services on RunxBuild deploy from a repository with runtime logs and metrics against a plan whose memory is explicit, and autoscaling is bounded by plans you pick rather than being open-ended. If you are sizing that service alongside a database, the RunxBuild hosting calculator shows them as separate line items.

Useful related references:

• What Does init Do in Python? It Sets Up Each Object, and self Is How
• Deploy a Java Backend for Free on RunxBuild
• How to Install Java on Ubuntu: Pick the Right JDK, Not Just Any JDK
• Services on RunxBuild

FAQ

Should I use Caffeine or Guava Cache?

Caffeine for new code — it outperforms Guava on both throughput and hit rate, with a similar API. Existing Guava Cache code works fine and is not worth migrating purely for its own sake.

Is a ConcurrentHashMap a cache?

No. It has no eviction, expiry, or size bound. It is fine for a small fixed set loaded at startup, but keyed on anything unbounded — a user id or search query — it grows until the heap is exhausted.

What is the difference between expireAfterWrite and expireAfterAccess?

expireAfterWrite bounds staleness: entries are discarded a fixed time after being written regardless of use. expireAfterAccess bounds idleness: a frequently read entry never expires, which is wrong if the underlying data changes.

Why does my cache work locally but not in production?

Almost always because production runs multiple instances, each with its own in-process cache. Explicit invalidation on write only affects the instance that handled the write; the others serve stale values until their TTL expires.

How do I know if a cache is worth keeping?

Enable statistics and check the hit rate. Below about 50% the cache is mostly overhead, and if the underlying lookups are cheap it may be negative value. Removing an ineffective cache is a legitimate optimisation.