Market Index technical details

Refresh schedule

• Market open: Weekdays from 9:35 AM Eastern. It loads the staging spreadsheet, then updates prices. On Mondays it first refreshes the S&P 500 company list.
• Market close: Weekdays from 5:00 PM Eastern, after closing prices have settled. It loads the staging spreadsheet again, then updates prices.
• DNS scores: Every day from midnight UTC, a little at a time, until every company has been scored that day.

The work runs as a queue of small steps, about one a minute, that each continue where the last one stopped. A step that fails is retried on its own, so a temporary outage delays an update rather than losing it.

Ingest pipeline

1. Constituents: The current S&P 500 list is read from the constituents table on Wikipedia. New companies are added, and companies that have left the index have their membership closed with the date they left. This runs automatically every Monday morning and can also be run from the admin panel.
2. Company domains: Each company is mapped to a primary domain. A domain embedded in the name (such as amazon.com) is used directly. Otherwise legal suffixes are stripped and the name becomes a .com, with the ticker as a last resort. An inferred domain is a guess: it is not checked against the company's own filings, so it can belong to someone else. Admins correct wrong domains by hand; corrected domains are scored straight away and are never replaced by later re-inference.
3. Prices: Prices are taken from the first source that returns consistent data: a Google Sheet staging table refreshed within an admin-set window (48 hours by default), then Massive.com, then Alpha Vantage. Prices are delayed and are not suitable for trading.
4. DNS scores: Each company domain uses the shared Standard DNS checks and findings, then the homepage, trust center, privacy and blocklist checks, graded with the Security Posture grade and the active scoring model from nine of its cards: Email Security, DNS Security, TLS and Certificates, Website Application Security, Endpoint Security, IP and Asset Reputation, Human and Social Engineering Exposure, Trust Center and Privacy. Brand Protection, Attack Surface, Network Security, Vulnerability Management and Credential Exposure take far more requests or paid services, so they aren’t checked here, and the Security Posture page can grade more areas than the score stored here. Domain Visualizer displays records and a graph.
5. Cache refresh: The page cache is cleared at the end of each run so the next visitor sees the new data.

How DNS scores stay reliable

• Rotating batches: Each run scores only a slice of the index, starting where the previous run stopped, so every company is refreshed over several runs instead of all at once.
• Modest concurrency: Each minute, up to ten domains are scored in parallel, each in its own run that makes at most 50 lookups. Higher rates would risk resolver throttling.
• Failure guard: A new score is stored only when every lookup it depends on was answered. A lookup that times out or fails in transport would look like a missing record, so if any lookup fails the new score is discarded and the company keeps its last verified score. A SERVFAIL counts as an answer only when a second resolver returns it too. Failed companies are retried after 10, 30 and 90 minutes, up to four attempts a day. The Higher Education, Healthcare and Non-Profit lists use the same scorer and the same rule, so scores are comparable across lists.
• Resolver comparison: The main lookups run on Cloudflare and are then repeated on Google, so answers that differ between the two show up as resolver-variance findings. Domain Visualizer makes this comparison only when a second resolver is selected.

Page caching

The Market Index API caches each page of results for 5 minutes during US market hours on weekdays and for 1 hour at other times. Results are served 100 companies per page.

Analyst notes

A low Market Index score identifies DNS protections or reliability checks that need investigation; it is not a statement about the company’s overall security. Verify that the domain belongs to the company. Open Domain Visualizer for records or Security Posture for remediation findings, checking the saved scan timestamp and cache state. Market Index stores the Security Posture grade from nine of its cards, leaving out Brand Protection, Attack Surface, Network Security, Vulnerability Management and Credential Exposure, so it can differ from the Security Posture page, which grades those too.