How to Reliably Scrape Official Steam Community Market

Scraping Valve's official Steam Community Market can begin with one free request. Its Market web routes are undocumented and are not an official API. Continuous catalog coverage needs target discovery, per-item scheduling, currency validation, failure classification, storage, and monitoring, with no published Market request allowance or stability contract.

Build the smallest useful pipeline#

A first version should move through four explicit stages:

1. Discover exact names from search/render and store identity records.
2. Schedule one endpoint and a small allowlist of items, not the entire catalog.
3. Save the raw response before parsing it into a typed observation.
4. Publish only observations that pass endpoint-specific identity, currency, and shape checks.

Keep those boundaries even in a small script. They let you replay a parser against saved responses without requesting Steam again, and they prevent an upstream HTML change from becoming a database full of zeroes.

The endpoint tutorials provide runnable collectors for catalog discovery, price summaries, raw listings, orderbooks, and sale history. Start with one, prove its success contract, then add another job.

Separate the collection jobs#

Do not make one loop responsible for every dataset. Give each upstream interface its own schedule and parser:

Different responses fail in different ways. A JSON orderbook, a grouped listing redirect, and an HTML page with missing embedded history should never share one generic success test.

Use one transport contract#

Centralize timeouts, HTML detection, and status classification while leaving payload validation in each endpoint parser. This Node.js 20 helper retries transient server failures, but it stops the run on an explicit rate limit or access denial. It never turns a failed request into an empty object.

// steam-http.mjs
const transientStatuses = new Set([502, 503, 504]);

export class SteamRequestError extends Error {
  constructor(message, details = {}) {
    super(message);
    this.name = "SteamRequestError";
    this.details = details;
  }
}

const wait = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

export async function fetchSteamJson(url, init = {}, attempts = 3) {
  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    let response;
    try {
      response = await fetch(url, {
        ...init,
        headers: { accept: "application/json", ...init.headers },
        signal: AbortSignal.timeout(20_000),
      });
    } catch (error) {
      if (attempt === attempts) {
        throw new SteamRequestError("Steam request failed", {
          kind: "network",
          cause: error.message,
          attempt,
        });
      }
      await wait(500 * 2 ** (attempt - 1) + Math.random() * 250);
      continue;
    }

    const raw = await response.text();
    if (response.status === 429) {
      throw new SteamRequestError("Steam rate limited the collector", {
        kind: "rate_limit",
        status: response.status,
        attempt,
        sample: raw.slice(0, 200),
      });
    }
    if (response.status === 403) {
      throw new SteamRequestError("Steam denied access", {
        kind: "access_denied",
        status: response.status,
        attempt,
        sample: raw.slice(0, 200),
      });
    }
    if (transientStatuses.has(response.status) && attempt < attempts) {
      await wait(500 * 2 ** (attempt - 1) + Math.random() * 250);
      continue;
    }
    if (!response.ok) {
      throw new SteamRequestError(`Steam returned HTTP ${response.status}`, {
        kind: "http",
        status: response.status,
        attempt,
        sample: raw.slice(0, 200),
      });
    }
    if (raw.trimStart().startsWith("<")) {
      throw new SteamRequestError("Expected JSON but Steam returned HTML", {
        kind: "unexpected_html",
        status: response.status,
        attempt,
        sample: raw.slice(0, 200),
      });
    }

    let payload;
    try {
      payload = JSON.parse(raw);
    } catch (error) {
      throw new SteamRequestError("Steam returned malformed JSON", {
        kind: "malformed_json",
        status: response.status,
        attempt,
        cause: error.message,
        sample: raw.slice(0, 200),
      });
    }
    return { payload, raw, status: response.status };
  }

  throw new SteamRequestError("Steam request exhausted its attempts");
}

Do not put a catch { return [] } around this function. The caller should record the typed error, leave the previous good observation intact, and give the target a later deadline. A catalog-wide rate limit should pause the job rather than trigger immediate requests from new addresses.

Define success for every response shape#

HTTP 200 is only a transport result. Steam can return an interstitial, a stripped page, a valid response for another grouped item, or a structurally valid object with the fields you need missing.

Define a storage boundary#

Raw evidence and parsed observations have different retention and query needs. A minimal relational design can keep them separate:

CREATE TABLE steam_raw_response (
  id              TEXT PRIMARY KEY,
  endpoint        TEXT NOT NULL,
  market_hash_name TEXT,
  requested_at    TEXT NOT NULL,
  http_status     INTEGER,
  outcome         TEXT NOT NULL,
  body            TEXT NOT NULL
);

CREATE TABLE steam_observation (
  endpoint         TEXT NOT NULL,
  market_hash_name TEXT NOT NULL,
  observed_at      TEXT NOT NULL,
  currency_id      INTEGER,
  payload_json     TEXT NOT NULL,
  raw_response_id  TEXT NOT NULL REFERENCES steam_raw_response(id),
  PRIMARY KEY (endpoint, market_hash_name, observed_at)
);

CREATE TABLE steam_target_state (
  endpoint         TEXT NOT NULL,
  market_hash_name TEXT NOT NULL,
  next_attempt_at  TEXT NOT NULL,
  consecutive_failures INTEGER NOT NULL DEFAULT 0,
  last_outcome     TEXT,
  PRIMARY KEY (endpoint, market_hash_name)
);

Do not place cookies, authorization data, proxy URLs, or other secrets in body or error samples. If raw responses are large, store compressed files or object blobs and keep only their content hash and location in the database.

Schedule from item demand#

Use exact market_hash_name values as targets. Assign faster intervals only to items that need them, spread initial requests across the interval, and place failed items back on a deadline queue after a delay. Sleeping inside a worker wastes the capacity needed by healthy items.

Steam does not document a safe request rate for Community Market pages. Treat HTTP 429 as a stop signal, not an invitation to rotate immediately and continue. Use bounded exponential backoff with jitter, cache responses, and cap retries. The documented Steam Web API allowance does not apply to these routes.

A deadline queue is safer than one sleep-heavy loop. Select targets whose next_attempt_at is due, cap the number of concurrent requests, and write the next deadline after each outcome. Give catalog discovery and full sale-history backfills long intervals; reserve shorter intervals for the small item set whose current price or depth actually matters.

Validate currency and identity#

Country and currency query parameters do not guarantee the response currency on every Market interface. Validate eCurrency or the relevant currency ID before publishing numeric values. The cs2.sh Steam collector accepts only currency ID 1 for its USD dataset.

Exit geography can affect the response. If an authorized collection deployment uses proxies for capacity, keep the region consistent and verify the payload rather than trusting the proxy label. Do not use proxying to disguise residence, evade restrictions, or continue through an explicit access denial; Valve's agreement separately restricts that behavior.

Grouped pages make identity equally important. Check the final URL, exact embedded history key, and resolved asset name before attaching data to a target.

Keep evidence for every failure#

Store raw responses before deriving prices or candles. Record HTTP status, content type, response shape, target, attempt, and collection time without logging cookies or proxy credentials.

Distinguish at least these outcomes:

• Valid JSON with data.
• Valid success=false for one unresolved item.
• Rate limiting or access denial.
• Marketplace server failure.
• HTML where JSON was expected.
• Currency mismatch.
• Malformed depth arrays.
• Listing page present but exact sale history absent.

A catalog-wide run can look healthy while producing no rows if monitoring counts only handled errors. Alert on freshness and successful publications by dataset.

Test parsers without calling Steam#

Keep one sanitized fixture for every accepted shape and every important failure shape. At minimum, test:

• A normal response with all optional fields present.
• A valid response with nullable fields absent.
• success=false.
• HTML returned where JSON is expected.
• An odd-length compact orderbook ladder.
• A listing whose asset reference is missing.
• A grouped page containing history for a different exact name.
• A currency mismatch.

Parser tests should read local fixtures and make no network requests. Add a separate, opt-in smoke test for the real endpoint if the project needs one. That split keeps unit tests deterministic and makes a changed live response evidence to inspect rather than an automatic fixture rewrite.

Before calling the collector production-ready, verify that a failed parse leaves the last good data untouched, every accepted row points back to raw evidence, a 429 pauses work, and freshness monitoring is based on published observations rather than attempted requests.

What cs2.sh replaces#

cs2.sh operates this collection and retained storage as one part of a broader service. It provides current sell listings and buy orders, OHLC candles at 5m through 1d, Steam orderbook snapshots, Steam sale history back to 2013, and data from BUFF, Youpin, CSFloat, Skinport, and C5Game.

Self-building is reasonable when Steam collection is part of the product. If it is only a prerequisite, the apparently simple free requests quickly become separate parsers, item schedules, IP capacity, cookies, response fingerprints, backfills, raw storage, and freshness alerts. cs2.sh provides current prices and historical prices without making that maintenance your application's problem.

Disclosure: this free DIY guide is published by cs2.sh. Review the Steam Subscriber Agreement before automating any Steam interaction. This operational article describes observed public interfaces; it does not authorize circumvention. Tutorial code and repository behavior were reviewed on August 25, 2026.