Metadata Fetcher
Look up a book in Google Books, Open Library and Project Gutenberg, rank the candidates by how well they match, and run it in Node or as a Docker HTTP service.
@prose-reader/metadata-fetcher fetches a book's metadata from online catalogs. archive-reader reads what the file itself carries; this reads what the world knows about it.
It takes a compact FetchMetadataInput, asks a list of pluggable providers, scores every rich catalog candidate against that input, and returns the ranked alternatives plus the per-provider detail behind them:
import { resolveArchive } from "@prose-reader/archive-reader"import { createGoogleBooksProvider, createOpenLibraryProvider, createProjectGutenbergProvider, fetchMetadata, metadataInputFromResolvedArchive,} from "@prose-reader/metadata-fetcher"
const resolved = await resolveArchive(archive)const input = metadataInputFromResolvedArchive(resolved)const fetched = await fetchMetadata(input, { providers: [ createProjectGutenbergProvider(), createGoogleBooksProvider({ apiKey: "your-api-key" }), createOpenLibraryProvider(), ],})
const match = fetched.matches.find(({ accepted }) => accepted)match?.metadata.cover?.uri // what the best accepted candidate saysmatch?.score // …and how sure we are it is this bookNothing here needs an archive: hand-built terms work just as well, which is what a "search by title" form or a filename-derived guess produces.
const fetched = await fetchMetadata( { title: "Dune", authors: ["Frank Herbert"] }, { providers: [createOpenLibraryProvider()] },)Lookup input
Section titled “Lookup input”The input is owned by the fetcher and contains exactly the fields its providers and matcher understand:
type KnownMetadataIdentifierScheme = | "ISBN" | "GTIN" | "DOI" | "GoogleBooks" | "OpenLibrary" | "ProjectGutenberg" | "UUID" | "URN" | "URL" | "Unknown"
type MetadataIdentifierScheme = | KnownMetadataIdentifierScheme | (string & {}) // custom catalog/application scheme
type MetadataIdentifier = { value: string scheme: MetadataIdentifierScheme}
type FetchMetadataInput = { title?: string authors?: ReadonlyArray<string> identifiers?: ReadonlyArray<MetadataIdentifier> series?: string publisher?: string publishedYear?: number languages?: ReadonlyArray<string> numberOfPages?: number}Some fields start a request (title, authors, series, identifiers); others disambiguate the candidates that come back (publisher, publishedYear, languages, page count). ISBN, GTIN and catalog ids all use the same identifier shape: { value: "9780441013593", scheme: "ISBN" } and { value: "zyTCAlFPjgYC", scheme: "GoogleBooks" }. The known schemes give TypeScript autocomplete, while any consumer-defined string remains valid. Unknown explicitly preserves a source identifier whose namespace cannot be classified; URL means a validated absolute HTTP(S) identifier; UUID and URN are what an EPUB's urn:uuid: unique identifier and other URNs resolve to. Reading a specific identifier back out of an input or a candidate belongs to archive-reader: identifierValue accepts both the ISBN and GTIN schemes for either, reads a catalog id out of the reference URL that carries it, and normalizes the authored value — that is what the providers and the matcher use. Fields with no input role — covers, descriptions, subjects, rights, layout and format-specific metadata — are deliberately absent.
metadataInputFromResolvedArchive is the explicit bridge from archive-reader. It only requires the metadata projection, so callers do not need to resolve reading order, TOC or sources for a lookup:
const resolved = await resolveArchive(archive, { include: ["metadata"] })const input = metadataInputFromResolvedArchive(resolved)The adapter selects authors from contributors, the first series, and the edition publisher/year with original-publication data as fallback. It forwards every identifier with its scheme and removes only archive-specific details such as unique.
The fetched entity
Section titled “The fetched entity”fetchMetadata returns a plain-JSON entity — structured-clone-able, persistable, cacheable — the metadata-fetching equivalent of ResolvedArchive:
type FetchedMetadata = { version: number /** every match across every provider, ranked best-first */ matches: MetadataMatch[] /** per-provider detail, keyed by provider id */ sources: Record<string, { provider: { id, name }; matches: MetadataMatch[] }> /** providers whose search threw */ /** providers that dropped out, with the HTTP status when there was one */ failedProviders: { id: string; status?: number }[]}version is the schema version, currently 5, bumped only when the shape or meaning of existing fields changes incompatibly — a MetadataMatch included. Every candidate uses the explicit publication.original / publication.edition vocabulary.
The same value is exported as FETCHED_METADATA_VERSION, for a consumer that persists part of the entity, or builds that shape without calling fetchMetadata, and still wants to tag what it stores:
import { DEFAULT_METADATA_MATCH_MIN_SCORE, FETCHED_METADATA_VERSION, scoreMetadataCandidate,} from "@prose-reader/metadata-fetcher"
// keep only the best accepted match, with the schema it was written inconst [best] = fetched.matches.filter(({ accepted }) => accepted)if (best) db.put(bookId, { version: FETCHED_METADATA_VERSION, match: best })
// a migration scoring records stored before metadata-fetcher, offline, writes// the same shape — and has no fetchMetadata result to read the version from,// so it accepts by the threshold fetchMetadata uses when given no minScoreconst { score, signals } = scoreMetadataCandidate(input, legacy.metadata)db.put(bookId, { version: FETCHED_METADATA_VERSION, match: { providerId: legacy.providerId, score, signals, accepted: score >= DEFAULT_METADATA_MATCH_MIN_SCORE, metadata: legacy.metadata, },})matches are deliberately not consolidated into a single metadata record. A score says how strongly a candidate resembles the input; it does not prove that several accepted candidates describe the same edition. Keeping the candidates separate avoids silently mixing fields from competing records and leaves selection policy with the consumer.
sources groups those same matches by provider. Unlike archive sources, the key space is open: providers are pluggable, so the keys are whatever providers you passed.
Matches: how it matched
Section titled “Matches: how it matched”A match is a candidate plus the evidence for it:
type MetadataMatch = { providerId: string /** 0–1 aggregate confidence */ score: number /** every field comparison behind the score */ signals: { field: "isbn" | "gtin" | "identifiers" | "title" | "authors" | "series" | "publisher" | "publishedYear" | "languages" | "numberOfPages" score: number weight: number /** the two values compared, rendered for display */ query: string candidate: string }[] /** reached minScore */ accepted: boolean metadata: ResolvedMetadata id?: string url?: string raw?: unknown}Scoring is done by the package, identically for every provider — a provider never grades its own homework, so scores stay comparable across catalogs and a match stays explainable to a user ("same title and author, different edition publisher"):
fetched.matches[0]?.signals// [// { field: "title", score: 1, weight: 0.8, query: "Dune", candidate: "Dune" },// { field: "authors", score: 1, weight: 0.5, query: "Frank Herbert", … },// { field: "publisher", score: 0.2, weight: 0.15, query: "Ace", candidate: "Chilton Books" },// ]The rules:
-
Only comparable fields count. A field is compared when both sides state it. What the book doesn't know can neither raise nor lower a candidate, so a sparse EPUB and a rich one land on the same 0–1 scale. A candidate with nothing in common to compare scores
0. -
The aggregate is a weighted average of those fields, with weights expressing intent rather than statistics (exported as
METADATA_MATCH_WEIGHTS):Field Weight isbn,gtin1 identifiers are the book title0.8 what a human recognizes it by identifiers0.6 other identifier systems, compared within the namespace both sides state authors0.5 compared as people series0.3 publishedYear0.2 year proximity against the candidate's publication data publisher,languages,numberOfPages0.15 publication details — they must never sink a convincing match alone -
Confirmed identity settles it. An agreeing ISBN or GTIN pins the score to
1whatever else disagrees. A contradicting ISBN or GTIN scores0, which sinks a plausible-looking wrong edition. ISBN-10 and ISBN-13 of the same book compare equal (toIsbn13is exported). -
Other identifiers are compared within a namespace both sides state. A catalog id is read the way
identifierValuereads it, so a Gutenberg URL and aProjectGutenbergnumber, or a Google Books link and aGoogleBooksvolume id, are one namespace; any other stated scheme is its own. A UUID is read the same way, so its bare andurn:uuid:spellings, in either case, are one value, and any other URN is scoped by the namespace it names and compared as RFC 8141 compares URNs, whatever the case of that namespace or of a percent-encoded octet: aurn:nbn:and aurn:issn:identifier are never compared. Ids compare in the spelling their namespace canonicalizes to, so a Google Books volume id is case-sensitive, while a DOI compares without case, as the DOI system defines it. A shared identifier pins the score to1. A different id in the same namespace is a disagreement, scored0at its weight rather than pinned, since one edition can sit under several records of a catalog. Lists with no namespace in common are not compared at all, and add noidentifierssignal: an EPUB 3urn:uuidand a catalog's own id are different id spaces, not a contradiction. An untyped value that is neither a URN nor a known catalog's URL (anUnknownstring, a publisher's web page) has no namespace, and nor has any value whose scheme is blank, a URN included, so neither is ever compared, even against an equal value of another scheme. -
Comparisons are fuzzy where the world is. Titles compare on character bigrams, against both the candidate's main title and that title composed with its
subtitleentry — a book statesDune: A Novelin onedc:titlewhere a catalog splits it in two, and only the comparison string is composed, never the returned metadata. The subtitle asymmetry is repaired on top (Dune≡Dune: a novel, butDune: Book One≠Dune: Messiah). An explicit conflicting volume, part, or book number makes the title score0(Vol. I≠vol. 2) — in any compared form, so a bareIrinacannot launder the candidate's ownVol. 1against a query forVol. 2, and a number the query states in its title is also compared against the candidate's seriesposition, where a catalog keeps the same fact. Names compare on their token set (Herbert, Frank≡Frank Herbert); diacritics and punctuation are folded (Les Misérables≡Les Miserables); languages compare on their primary subtag (en-US≡en). -
Flat publication evidence uses the best candidate value.
publisherandpublishedYearcompare against both the candidate's original and edition publication details when present; the closest value supplies the single signal. -
A rating is never evidence.
aggregateRatingis returned but deliberately not scored: how much a crowd liked a book says nothing about whether it is this book, and a popular edition must not outrank the right one.
minScore (default 0.5, exported as DEFAULT_METADATA_MATCH_MIN_SCORE) decides which matches are accepted. Rejected matches are kept and ranked, not dropped: "the catalog found three books, none convincing" is an answer, and it is exactly what a "did you mean?" picker renders.
if (!fetched.matches.some(({ accepted }) => accepted)) { showPicker(fetched.matches) // nothing convincing — let the user choose}Options
Section titled “Options”const fetched = await fetchMetadata(input, { providers: [ createProjectGutenbergProvider(), createGoogleBooksProvider({ apiKey: "your-api-key" }), createOpenLibraryProvider(), ], limit: 5, minScore: 0.5, includeRaw: false, signal: controller.signal,})| Option | Default | |
|---|---|---|
providers |
— | the catalogs to ask, queried concurrently. Their order is the tie-break precedence: equally-scored matches rank in declaration order |
limit |
5 |
hard cap on the matches kept per provider, best-scoring first, and the page-size hint passed to each provider |
minScore |
0.5 |
score a match must reach to be accepted; the default is exported as DEFAULT_METADATA_MATCH_MIN_SCORE, for a consumer scoring candidates itself with scoreMetadataCandidate |
includeRaw |
false |
keep each provider's own record on match.raw. Off by default: provenance most consumers don't need, and it can dwarf the normalized entity they persist |
signal |
— | cancellation, forwarded to every provider |
Error policy
Section titled “Error policy”Mirrors resolveArchive: a provider that throws — a rate limit, an outage, a network error, a malformed response — never fails the fetch. Providers are asked concurrently and each is isolated, so one failing costs you that catalog and nothing else. It is logged through the debug Report and listed in failedProviders, the only trace left:
// don't persist "we found nothing" when we simply couldn't askif (fetched.failedProviders.length === 0) cache.set(bookId, fetched)
// rate limited rather than broken: worth asking again, laterconst throttled = fetched.failedProviders.filter(({ status }) => status === 429)Each entry carries the catalog's HTTP status when the failure was a response — 429 (ask again later), 5xx (the catalog is down), 4xx (we asked wrong) — and no status when it wasn't one, which is itself the answer: a network error, an unparseable payload, a bug. Three states are distinguishable: a provider in sources answered, one in failedProviders threw, one in neither was never asked (narrowed out by the providers option).
fetchMetadata itself does not retry or interpret Retry-After; providers own that policy because only they know which operations are idempotent and which failures their catalog treats as transient. The built-in providers share one:
- Each request makes at most three attempts.
- A network failure, a rate limit (
429) and an outage (500,502,503,504) are retried with abortable exponential backoff and jitter, bounded at 3 seconds per wait. Cancelling throughsignalinterrupts a wait. - When the catalog sends
Retry-After, the provider waits exactly that long instead. ARetry-Afterbeyond the 3-second bound is not waited out: the provider gives up at once, since an earlier retry would only be refused again. - Client errors, such as an invalid key, are not retried.
A provider that exhausts its retries, or gives up on a long Retry-After, is reported here normally, with its status.
Cancelling through signal is the one exception: it rejects, since the caller asked for it.
Choosing metadata
Section titled “Choosing metadata”The globally ranked matches list makes the simplest policy explicit: select its first accepted entry. A UI can instead present several alternatives, prefer a provider, or require a confirmed identifier before applying anything.
const match = fetched.matches.find(({ accepted }) => accepted)
if (match !== undefined) { importMetadata(match.metadata)}Each candidate remains a ResolvedMetadata, so it can be handled with the same application code as metadata read from the publication. The fetcher does not decide how local and remote values should be reconciled.
aggregateRating is each provider's own crowd, normalized to the shared 0–5 scale — Google's mean and Open Library's are different populations of readers, not two measurements of one number, so a picker showing both should attribute them (match.providerId) rather than average them. It is also the most volatile thing in a persisted FetchedMetadata: a title, an ISBN and a page count do not move, a rating does. Refresh it on its own schedule, or treat a cached one as indicative.
Providers
Section titled “Providers”| Provider | Import | Catalog | Notes |
|---|---|---|---|
createProjectGutenbergProvider |
@prose-reader/metadata-fetcher |
Project Gutenberg | official per-eBook RDF; exact Gutenberg identifiers only, no key |
createGoogleBooksProvider |
@prose-reader/metadata-fetcher |
Google Books | broad edition metadata and covers; API key required |
createOpenLibraryProvider |
@prose-reader/metadata-fetcher |
Open Library | free, no key; books broadly, comics and manga much less so |
Project Gutenberg
Section titled “Project Gutenberg”import { createProjectGutenbergProvider } from "@prose-reader/metadata-fetcher"
const provider = createProjectGutenbergProvider({ userAgent: "MyReader/1.0 (contact@example.com)",})| Option | Default | |
|---|---|---|
baseUrl |
https://www.gutenberg.org |
catalog origin; override for a mirror or stub |
fetch |
the global one | for tests, a custom agent, or a caching layer |
userAgent |
— | optional identifying user agent |
Lookup strategy, exactly one request, retried under the built-in providers' retry policy: the provider recognizes an official Gutenberg URL in identifiers (including /ebooks/78139, /files/78139/…, /cache/epub/78139/…, and the older /78139 form), or a numeric identifier whose scheme is ProjectGutenberg. It then requests Gutenberg's official per-eBook RDF record at /cache/epub/{id}/pg{id}.rdf. A title or author alone never triggers the provider, so it neither crawls the website nor returns a fuzzy Gutenberg hit. Arbitrary URLs and identifiers with another authored scheme are ignored.
The confirmed candidate carries { value: "78139", scheme: "ProjectGutenberg" }, which the matcher compares with the book's URL as the same Gutenberg id: the match is decisive even when the catalog and embedded titles differ. A missing RDF record returns no candidate; a failing or malformed response is reported through failedProviders like any other provider failure.
Mapping (per-eBook RDF → ResolvedMetadata; same exhaustive-destructure check as the other providers):
| Project Gutenberg RDF | Resolved | |
|---|---|---|
pgterms:ebook@rdf:about |
identifiers |
numeric scheme ProjectGutenberg, which the matcher compares with a Gutenberg URL the book states |
dcterms:title |
title, titles |
|
dcterms:creator, marcrel:* |
contributors |
common MARC relators normalize to roles such as author, translator, editor, and illustrator; unknown codes remain verbatim |
dcterms:issued |
publication.edition.date |
Gutenberg ebook release date, parsed through day precision when available; it is not the work's original publication date |
dcterms:publisher |
publication.edition.publisher |
the publisher of the Gutenberg edition |
pgterms:marc260 |
publication.original.date, publication.original.publisher |
original print statement; only one unambiguous $c year and $b publisher are promoted |
dcterms:rights |
rights |
|
dcterms:language |
languages |
RDF values are already BCP 47 |
LCSH dcterms:subject, pgterms:bookshelf |
subjects |
deduped and capped at 25; Library of Congress classification codes are excluded |
pgterms:marc520, then dcterms:description |
description |
catalog summary preferred over a general note |
medium/small cover in dcterms:hasFormat |
cover |
medium preferred; absolute HTTP(S) url, confidence: "derived" |
The response is parsed and normalized in memory. The provider has no database, cache, file writes, or other persistence.
Google Books
Section titled “Google Books”import { createGoogleBooksProvider } from "@prose-reader/metadata-fetcher"
const provider = createGoogleBooksProvider({ apiKey: "your-api-key",})| Option | Default | |
|---|---|---|
apiKey |
— | required application API key |
baseUrl |
https://www.googleapis.com/books/v1 |
API root; override for a stub |
fetch |
the global one | for tests, a custom agent, or a caching layer |
Lookup strategy, from most exact to broadest: an authored GoogleBooks identifier (or official Google Books/API URL), an ISBN identifier, then title plus first author. When the author makes a title query too narrow, the provider tries title alone. Search requests ask only for books, preserve Google's relevance order, and never exceed 40 results — the API's maximum. A query with none of these terms returns no candidates without making a request.
Title and author reach intitle:/inauthor: unquoted. A quoted phrase makes Google demand the catalog title contain it verbatim, which a title carrying a volume marker, a scanner tag or a translated subtitle never does — and with no query relaxation to retry with, the lookup would report nothing for a book Google holds. Unquoted, the catalog ranks loosely and the matcher decides, which is where precision belongs.
Unquoted also means Google parses the term as syntax rather than reading it literally, so anything in it that Google would take as an instruction is neutralized first: quotes and backslashes (they unbalance the query), colons (Dune inauthor:Asimov would search authors) and a hyphen starting a word (BLAME! -Master Edition- would demand books without "Master"). A term left with nothing searchable counts as absent, so a title that is only query syntax makes no request.
For the same reason a title search asks Google for four times limit: those candidates arrive in Google's relevance order rather than the matcher's, so the edition the caller described can sit below the ones Google finds most popular. fetchMetadata scores the wider page and trims to limit afterwards. Exact identifier and ISBN lookups need no such room and ask for limit.
To replace Oboku's googleVolumeId, pass it directly:
const input = { title: "Dune", identifiers: [{ value: googleVolumeId, scheme: "GoogleBooks" }],}An official Google Books website/API URL with scheme URL is recognized too. Every candidate carries its volume id as { value: id, scheme: "GoogleBooks" }, which the matcher compares with either spelling of the book's id, so an exact lookup is decisive, and metadataInputFromResolvedArchive can forward the id unchanged for a later refresh. A missing volume id falls through to ISBN/title when available. ISBN searches likewise retain the queried ISBN identifier even when a sparse Google record omits industryIdentifiers.
Each request follows the built-in providers' retry policy: rate limits and outages are retried, honouring Retry-After, and client errors such as an invalid key are not. When the last attempt fails, or when a successful exact response is malformed, the provider is reported through failedProviders normally.
Mapping (Google Books Volume → ResolvedMetadata; the resolver names every parsed field and asserts the rest is empty, so a field added to the parser fails the build until someone decides what becomes of it):
| Google Books | Resolved | |
|---|---|---|
id |
identifiers |
scheme GoogleBooks; also id on the match |
volumeInfo.title |
titles |
verbatim; the main title (first entry) |
subtitle |
titles |
its own entry, typed subtitle — never appended to the main title |
seriesInfo.volumeSeries |
belongsTo.series |
seriesId as a GoogleBooks identifier, orderNumber as position. Google never states the series name, and bookDisplayNumber / shortSeriesBookTitle are display strings by its own definition, so neither is mapped |
authors |
contributors |
role author |
publisher, publishedDate |
publication.edition |
partial dates retain the available year/month/day |
description |
description |
Google may provide simple HTML formatting |
industryIdentifiers |
identifiers |
ISBN schemes normalize to ISBN; every announced identifier is retained |
pageCount |
numberOfPages |
|
averageRating, ratingsCount |
aggregateRating |
Google's mean is already on the shared 0–5 scale (it rounds to a half star); a count alone states no rating, and a mean over zero ratings is dropped |
categories |
subjects |
deduped and capped at 25 |
language |
languages |
Google's best ISO 639-1 language |
imageLinks |
cover |
the largest linked size, or the thumbnail asked at 1,280 px wide; HTTPS, without the page curl. See Covers |
canonicalVolumeLink, then infoLink |
match url |
upgraded to HTTPS; a stable Google Books URL is synthesized from id when absent |
Google-specific values without a cross-format home (access and sale data, reading-mode flags, and so on) remain available on match.raw when includeRaw is enabled.
Covers
Section titled “Covers”A Google Books match carries the largest cover the provider can link without another request, and without the page curl Google draws on its covers.
Google's search links only two small sizes, at most about 128 px wide. A volume's own record also links large (800 px wide) and extraLarge when Google holds the publisher's cover, but only a lookup by GoogleBooks id reads that record. So the provider picks:
extraLargeorlarge, when the record links them.- Else
thumbnail, withfife=w1280added. Google's image server then serves the same cover at most 1,280 px wide. - Else the largest other link.
Each link goes over HTTPS and loses edge=curl, which serves the same image at the same size without the curl. The provider keeps every other parameter as Google wrote it. In particular, it never changes zoom: for a volume Google knows only from a library catalog, another zoom serves Google's "image not available" picture, or another page of a scanned book.
The fife size has these side effects, which no API reports:
- A publisher cover found by search stops at 1,280 px wide. A smaller source is enlarged to that width.
- A catalog-only volume has no larger image, so its thumbnail comes back enlarged to about 300 px, with no new detail.
- A scanned library book can show its title page instead of the cover its thumbnail shows.
- Google documents none of
fife,zoomoredge. If Google changes how it serves them, covers can come back small or wrong.
Open Library
Section titled “Open Library”import { createOpenLibraryProvider } from "@prose-reader/metadata-fetcher"
const provider = createOpenLibraryProvider({ userAgent: "MyReader/1.0 (contact@example.com)",})| Option | Default | |
|---|---|---|
baseUrl |
https://openlibrary.org |
API origin |
coversBaseUrl |
https://covers.openlibrary.org |
cover service origin |
fetch |
the global one | for tests, a custom agent, or a caching layer |
userAgent |
— | Open Library's API etiquette asks for an identifying one (app name + contact) and throttles anonymous traffic harder |
Lookup strategy, at most four requests: an ISBN search when the book states one — the catalog then verifies the identity for us; an exact id_project_gutenberg search when an identifier is an official Project Gutenberg URL or a numeric ProjectGutenberg identifier; then a precise title (+ first author) search when those identifiers are unknown to Open Library or absent. When that precise search is empty, a free-text title-and-author query handles records whose work title and subtitle were indexed separately. The Gutenberg crosswalk is this provider's own, because id_project_gutenberg is a field of Open Library's search index. A query with none of those terms yields no candidates rather than a fishing expedition. Each request follows the built-in providers' retry policy. Open Library throttles anonymous and fast traffic with 429, which the policy retries.
Mapping (search.json doc → ResolvedMetadata; same exhaustive-destructure check as the other providers):
| Open Library | Resolved | |
|---|---|---|
title |
titles |
verbatim; the main title (first entry) |
subtitle |
titles |
its own entry, typed subtitle — the matcher composes the two when it compares, so nothing is gained by joining them here |
author_name |
contributors |
role author |
first_publish_year |
publication.original.date.year |
|
language |
languages |
MARC 21 → BCP 47 (eng → en); unknown codes pass through, und/mul/zxx are dropped |
subject |
subjects |
capped at the first 25 — a popular work carries hundreds, most of them long-tail noise |
number_of_pages_median |
numberOfPages |
|
ratings_average, ratings_count |
aggregateRating |
the catalog's raw 0–5 mean of the work's ratings, unrounded. ratings_sortable is deliberately not mapped: it is Open Library's internal ranking key, not a rating it states |
cover_i |
cover |
absolute url on the cover service, confidence: "derived" |
id_project_gutenberg |
identifiers |
scheme ProjectGutenberg, which the matcher compares with a Gutenberg URL the book states |
key |
identifiers |
scheme OpenLibrary, e.g. /works/OL893415W; also id and url on the match |
An ISBN identifier is added only when the search was an ISBN lookup, and then it uses the queried value: the API answered "this work has that ISBN", which is a fact about the record. A title-search hit describes a work, whose editions each have their own ISBN, so picking one would be fabrication.
Only the mapped fields are requested (fields=), which is the difference between a few hundred bytes per hit and a few hundred kilobytes.
Running it as a service
Section titled “Running it as a service”A Docker image wraps this package in a small HTTP API, so metadata lookups don't have to happen inside your JavaScript app — and so you can try a provider against a real catalog with curl:
For repository development, copy apps/metadata-fetcher-api/.env.example to apps/metadata-fetcher-api/.env, set GOOGLE_BOOKS_API_KEY, then run pnpm run start:metadata-fetcher from the repository root. Compose loads that file automatically and it remains ignored by Git.
docker run -p 6382:6382 \ -e GOOGLE_BOOKS_API_KEY="your-api-key" \ -e OPEN_LIBRARY_USER_AGENT="MyApp/1.0 (me@example.com)" \ mbret/prose-metadata-fetcher-api:latest
curl "http://localhost:6382/metadata?isbn=9780441013593"The image is published with every release, tagged with the version and latest, for linux/amd64 and linux/arm64.
| Route | |
|---|---|
GET /health |
liveness, plus the providers the deployment exposes |
GET /metadata?title=&author=&isbn=… |
human-friendly lookup |
POST /metadata |
body is a FetchMetadataInput, options on the query string |
Both metadata routes answer with the FetchedMetadata entity verbatim. The options above (limit, minScore, includeRaw, plus a providers filter) are query parameters, and the deployment defaults are environment variables.
The service does no per-client rate limiting: who a client is, which address is real and how often each may call depend on your infrastructure and your own provider quotas, so that limit belongs to the proxy or gateway in front of it. It does bound its own capacity — MAX_CONCURRENT_LOOKUPS (default 32) caps the lookups one process runs at once, and a lookup beyond it answers 503 with Retry-After. This package has no limit of its own either. A built-in provider retries a catalog's 429 briefly, as its retry policy describes; one that persists is reported in failedProviders, and backing off further is the caller's decision.
For development, pnpm run start:metadata-fetcher starts the same image with the source bind-mounted: editing the express app or this package restarts it — node runs both as TypeScript, so nothing is rebuilt. It also serves a playground at / — enter metadata manually or choose an EPUB/CBZ/ZIP file, then inspect each candidate, its score and the signals behind it. In both cases the playground posts the canonical FetchMetadataInput JSON to /metadata, so the browser's network inspector shows the exact input an SDK consumer would provide. Uploaded files have no application-level size limit: archive-reader resolves them in memory and discards them before that compact input is posted, with no disk or browser storage. The playground is development-only: the production image sets NODE_ENV=production, and the route is then never registered.
See the app's README for the full reference.
Writing a provider
Section titled “Writing a provider”A provider is three things: a stable id, a display name, and a search returning normalized candidates. It never scores its own results — that stays in the package, identically for everyone.
import { type MetadataProvider } from "@prose-reader/metadata-fetcher"
const myProvider: MetadataProvider = { id: "myCatalog", name: "My Catalog", search: async (input, { limit, signal }) => { if (input.title === undefined) return []
const response = await fetch( `https://example.com/search?q=${encodeURIComponent(input.title)}` + `&author=${encodeURIComponent(input.authors?.[0] ?? "")}`, { signal }, ) const { results } = await response.json()
return results.slice(0, limit).map((result) => ({ id: result.id, url: `https://example.com/book/${result.id}`, raw: result, metadata: { title: result.name, identifiers: [{ value: result.isbn, scheme: "ISBN" }], contributors: result.authors.map((name) => ({ name, roles: ["author"], })), }, })) },}The asymmetry is intentional: providers receive the small FetchMetadataInput and return rich ResolvedMetadata candidates. The input exposes only lookup and matching evidence; results retain covers, descriptions, subjects, contributors and every other useful catalog value.
Helpers cover archive conversion and common input checks:
import { hasSearchTerms, metadataInputFromResolvedArchive,} from "@prose-reader/metadata-fetcher"
const input = metadataInputFromResolvedArchive(resolved)
// is there anything to go on at all? publication details or a page count// narrow a search but cannot start one, so they don't countif (!hasSearchTerms(input)) returnRules of thumb:
-
Return an empty list, don't throw, when the input carries nothing you can search on. Throwing is for actual failures — they land in
failedProviders. -
Throw
MetadataProviderResponseErrorwhen the catalog answers with a failing status, so the status reaches the caller instead of dying in a log line. Any error carrying a numericstatusworks too, which covers HTTP clients that throw their own:import { MetadataProviderResponseError } from "@prose-reader/metadata-fetcher"if (!response.ok) {throw new MetadataProviderResponseError(response.status, "My Catalog search failed")} -
Use
retryWithBackoffwhen an idempotent provider request has known transient failures.attemptscounts the initial call, and cancellation interrupts the next delay. Pass the same signal to the request when it should cancel an active call too. The helper deliberately leaves the retry predicate to the provider.requestedDelayMslets a failure name its own delay, such as a rate limit'sRetry-After, in place of the backoff; a delay longer thanmaxDelayMsrethrows at once instead of retrying early.parseRetryAfterHeaderreads the header, the error carries it asretryAfterMs, andresponseErrorRetryAfterMsreads it back from any error carrying one. Cancel the body of a response you do not read: Node'sfetchkeeps its connection busy until garbage collection, and failed attempts would pile up sockets:import {MetadataProviderResponseError,parseRetryAfterHeader,responseErrorRetryAfterMs,responseErrorStatus,retryWithBackoff,} from "@prose-reader/metadata-fetcher"const response = await retryWithBackoff(async () => {const response = await fetch(url, { signal })if (!response.ok) {await response.body?.cancel()throw new MetadataProviderResponseError(response.status, "My Catalog search failed", {retryAfterMs: parseRetryAfterHeader(response.headers.get("Retry-After")),})}return response},{attempts: 3,initialDelayMs: 1_000,maxDelayMs: 5_000,signal,shouldRetry: (error) => [429, 503].includes(responseErrorStatus(error) ?? 0),requestedDelayMs: responseErrorRetryAfterMs,},) -
Forward
signalto every request you make. -
Treat
limitas a page-size hint; the fetch caps the ranked result anyway. -
Normalize honestly. Populate a field only when the record actually states it — a fabricated value is a signal the matcher will score, and a wrong
ISBNidentifier is the single most damaging thing you can emit. -
rawis yours: put your parsed record there for provenance and provider-specific fields with no home in the vocabulary. It is kept only when the consumer asks for it.
The matching primitives are exported too (scoreMetadataCandidate, titleSimilarity, personNameSimilarity, textSimilarity, normalizeForComparison) if you want to rank something of your own by the same rules — scoreMetadataCandidate(input, candidate) takes a FetchMetadataInput and a rich ResolvedMetadata candidate. Accept by DEFAULT_METADATA_MATCH_MIN_SCORE to draw the line where fetchMetadata draws it by default.