Keeping a homebuyer app working when the ERP behind it goes down

We already had a cache for the parts of Sapphire every buyer shares. Since December 2025, communities, floor plans, lots and the public home listings have sat in Redis for 24 hours, keyed on the query’s parameters. That works because a floor plan looks the same to everyone.

A buyer’s milestones don’t, and the Acting-As header was the only thing keeping one buyer’s data apart from another’s. Several home-scoped endpoints had no local check that the home ID belonged to the caller. Put a cache in front of that call keyed on the home ID, and on a cache hit the header is never sent. Buyer B asks for a home ID buyer A already warmed and gets buyer A’s record back. Nothing throws, and testing with one account would never show it.

So the GitHub issue asked for research and a design doc first, and the doc was reviewed before anyone wrote code. It set two rules. Every per-buyer key carries the buyer’s identity, built by one helper so no call site writes a key by hand:

sapphire:buyer:<entity>:<buyerKey>:<paramsHash>

The buyerKey is the first 16 hex characters of a SHA-256 hash of the E.164 phone, so “(801) 555-1234” and “+18015551234” land on the same key and no raw phone number ever sits in Redis. It’s a separate segment, apart from the parameter hash, so you can see it in a key listing and clear one buyer’s data with a single scan.

The second rule was a local ownership check on every home-scoped endpoint, so the cache key is never the only thing standing between two buyers.

The healthy path didn’t change at all. Every request still goes to Sapphire live, so nothing got faster and nothing got staler.

fetchWithSnapshot wraps the five buyer reads: homes, messages, milestones, home details and profile. When Sapphire answers, the response is written to that buyer’s snapshot key with a seven-day TTL, alongside a cachedAt time, and returned with stale: false. When the Sapphire call throws, it reads the snapshot and returns it with the original cachedAt and stale: true. With no snapshot, it rethrows and the screen shows the same error it always did.

Since reads are live, the seven days only caps how old an outage fallback can be.

Sapphire also fails in a quieter way. It often returns a 200 with field-level errors and part of the data, and for buyer homes we’d rather show 79 good homes than throw them out over one bad field. Those partial answers still go to the buyer, but they’re never written to the snapshot. The same goes for the updates feed, where Sapphire’s 403 for a buyer with no linked homes is mapped to an empty list. A flapping upstream can’t overwrite a good fallback with a broken one.

The cache fails open too. If Redis is down, reads fall through to Sapphire as if the cache didn’t exist, and write errors are logged and swallowed. Redis commands on this client time out after three seconds, because a connection that stays open but stops answering would otherwise leave a fallback read hanging with no limit.

We chose Redis alone over Postgres or both. The fallback only has to last from the last good read to the outage, and seven days covers that. A Postgres copy would have put homeowner data in our main database permanently. The cost we took on is that a Redis restart empties every snapshot, and the first outage after one has nothing to fall back on.

A code review on the cache PR caught a gap, and when we checked it, it was wider than reported. Nothing bounded a Sapphire call that simply hung. The only limit was the Node HTTP client’s default of about 300 seconds, and the app’s TanStack Query retries only fire when a request fails, never while it’s waiting. So during a hang, a buyer watched a spinner for up to five minutes before the snapshot could help.

Every Sapphire call now carries its own deadline:

• 15 seconds for JSON reads, so a fallback is served quickly
• 60 seconds for base64 image payloads
• 30 seconds for writes like service requests, which have no fallback and can carry photo attachments

The deadline is an AbortSignal created fresh for each request. The catalog and construction clients are shared singletons, and a signal set once on the constructor would fire once and then abort every request after it for the life of the process, so each request gets its own.

We didn’t add server-side retries. The app already retries three times with backoff, and stacking more on top would have multiplied the load on a server that was already struggling.

A buyer should know when they’re looking at saved data. The buyer endpoints now return a sapphireMeta field with lastUpdatedAt and stale, and when stale is true, the screen shows a small notice above the data: “Showing saved data from 2 hours ago. It will refresh automatically.” When Sapphire comes back, the normal refetch replaces the data and the notice goes away by itself.

It’s on the my homes list, the updates feed, the milestone tracker and home details in the app, and on the same screens in the buyer website, which has no updates feed.

Two details took more care than the banner did. The API deploys the moment it merges, but the app updates over the air later, so old app versions keep calling the API for a while. The new field is additive, and for milestones, which returned a bare array that old versions parse, we added a separate getMilestonesWithMeta endpoint and left the old one alone until every shipped version is off it. The updates feed is built from two Sapphire calls, messages and the buyer’s homes, so the notice reports the older of the two snapshots. Otherwise a days-old home list could sit under a fresh-looking feed with no warning.

The relative time (“2 hours ago”) uses Intl.RelativeTimeFormat, which Hermes on iOS doesn’t reliably support. Building it at the top of a shared module would have crashed every screen that shows the banner, so construction is guarded and falls back to plain English.

A saved home list is also a record of who owns what, so it can’t outlive a change. Linking a home clears the buyer’s homes snapshot. Unlinking clears homes, home details and milestones, so a stale snapshot can’t keep authorizing a home the buyer just removed. A new service request clears home details and messages, and deleting an account, from the app or from either admin path, clears everything for that buyer. Each clear is best effort, so a cache problem never turns a successful unlink into an error the buyer might retry.

That left a race. A read that started just before an unlink could finish just after the clear and write the old list back. Clearing now writes a short-lived marker for the buyer first, and snapshot writes are skipped while it exists. It lasts 20 seconds, which is the 15 second read deadline plus a five second margin, so it outlives any read that could have started before the clear. The marker is only checked on the write, which already happens off the response path, so the buyer’s request does no extra Redis read.

The ownership check, verifyHomeOwnership, runs before home details, milestones, photos, photo content and service requests. A home in the buyer’s verified list goes through. A home that’s verifiably not theirs returns not found, so the response doesn’t reveal whether that home ID exists. If ownership can’t be confirmed, because Sapphire is down and there’s no snapshot, it fails closed with a server error. The check reads the buyer’s homes through the snapshot cache with a 120 second window, so it adds no Sapphire call per request and keeps working during an outage. It also refuses partial home lists, since a home that failed to load would look like a home the buyer doesn’t own.

Logging out doesn’t clear anything on the server. Every read still requires a signed-in session, and the same buyer may be signed in on another phone. The copy the app keeps on the device, in MMKV, is wiped on logout and on account switch.

We left construction photos, message attachments and avatars off the server cache. Each is a multi-megabyte base64 payload, and putting them in the same Redis instance that backs rate limiting wasn’t worth it when the app already keeps viewed photos on the device for a day. The logs record how often photo requests fail during outages, and we’ll revisit it if those numbers say to. A buyer who has never opened a screen before an outage still gets the error card, since there’s nothing saved to show.

The cache PR added 36 tests, including two buyers asking for one home ID while Sapphire is down, a check that no raw phone number appears in any key, and the poison guards on each surface. The ownership check added 13 more. The cache, the deadlines, the ownership check and the notice went out together in the July 23, 2026 release, so no version ever ran the cache without the ownership check behind it. The marker fix followed on August 10.