[{"data":1,"prerenderedAt":1996},["ShallowReactive",2],{"doc:\u002Fdocs\u002Fapi-datasource":3,"docs-nav":1771},{"id":4,"title":5,"body":6,"description":1762,"extension":1763,"meta":1764,"navigation":1766,"path":1767,"seo":1768,"stem":1769,"__hash__":1770},"docs\u002Fdocs\u002Fapi-datasource.md","HTTP APIs as datasources",{"type":7,"value":8,"toc":1736},"minimark",[9,13,36,52,57,150,165,170,193,204,211,215,352,356,363,654,664,668,686,734,738,842,849,868,872,959,966,970,997,1000,1007,1011,1021,1024,1028,1031,1035,1041,1084,1091,1102,1106,1211,1231,1237,1246,1252,1271,1285,1289,1301,1366,1398,1402,1418,1422,1442,1446,1456,1646,1649,1654,1658,1677,1681,1684,1688],[10,11,5],"h1",{"id":12},"http-apis-as-datasources",[14,15,16,17,21,22,25,26,29,30,35],"p",{},"An ",[18,19,20],"strong",{},"HTTP API connection"," reaches any REST\u002FJSON API — a financial data broker, a\nCRM, an internal service, a public open-data endpoint. A datasource ",[18,23,24],{},"«API call»","\nmakes one or more requests to it at each refresh, turns the records of the answers\ninto one table and keeps it as a snapshot, refreshed by hand or on a schedule like\nevery other database datasource. It is input only: flows read it, nothing is sent\nback to the API. It is created from a folder in Explore with ",[18,27,28],{},"New → Datasource from a\ndatabase",", choosing the API connection; the other sources are listed in\n",[31,32,34],"a",{"href":33},"\u002Fdocs\u002Fsources","Sources",".",[14,37,38,39,42,43,46,47,51],{},"Like the other connections it lives in a folder and needs ",[18,40,41],{},"CONNECT"," on that folder\nto be created, tested and used. ",[18,44,45],{},"Any address is allowed"," — public, private,\n",[48,49,50],"code",{},"localhost"," — as for databases and SFTP: CONNECT is the barrier, so give it to the\npeople who may make the workers call those addresses.",[53,54,56],"h2",{"id":55},"the-connection","The connection",[58,59,60,73],"table",{},[61,62,63],"thead",{},[64,65,66,70],"tr",{},[67,68,69],"th",{},"Field",[67,71,72],{},"Meaning",[74,75,76,88,122,138],"tbody",{},[64,77,78,82],{},[79,80,81],"td",{},"Base URL",[79,83,84,87],{},[48,85,86],{},"https:\u002F\u002Fapi.example.com\u002Fv1",": scheme, host, optional port and path. No user or password in it, no query string, no fragment: they belong to the authentication and to the datasource.",[64,89,90,93],{},[79,91,92],{},"Authentication",[79,94,95,98,99,102,103,106,107,110,111,98,114,117,118,121],{},[18,96,97],{},"None","; ",[18,100,101],{},"API key",", sent in a header or in a query parameter whose name you choose (",[48,104,105],{},"X-API-Key",", ",[48,108,109],{},"apikey","…); ",[18,112,113],{},"Bearer token",[18,115,116],{},"User and password"," (HTTP basic); ",[18,119,120],{},"OAuth2 client credentials"," (token URL, client id, client secret, optional scope; the client sent in the form body or as basic authentication).",[64,123,124,127],{},[79,125,126],{},"Default headers",[79,128,129,130,133,134,137],{},"Sent with every request to the API's host, one ",[48,131,132],{},"Name: value"," per line (",[48,135,136],{},"Accept: application\u002Fjson","). They are not secret: the API returns them to whoever can see the connection. Put credentials in the authentication.",[64,139,140,143],{},[79,141,142],{},"Test path",[79,144,145,146,149],{},"What «Test connection» calls, below the base URL (",[48,147,148],{},"\u002Fstatus","); empty = the base URL itself. The test fails on any HTTP status from 400 up, showing it with the start of the answer.",[14,151,152,153,156,157,160,161,164],{},"The secret — API key, token, password or client secret — is encrypted at rest with\nthe same Fernet key as every other connection secret (",[48,154,155],{},"SECURITY__FERNET_KEY","), travels\nencrypted to the worker and is decrypted only to be sent. The API never returns it\n(",[48,158,159],{},"has_password"," says whether one is stored). An OAuth2 token is asked to the token\nURL, kept in the worker until it expires (30 seconds early), and asked again ",[18,162,163],{},"once","\nwhen the API answers 401 — a second 401 fails the refresh.",[166,167,169],"h3",{"id":168},"credentials-stay-on-the-apis-host","Credentials stay on the API's host",[14,171,172,173,176,177,180,181,184,185,188,189,192],{},"A request carries the credentials ",[18,174,175],{},"and"," the headers of the connection and of the\ncall only when it goes to the origin of the base URL — the same scheme, host and port.\nA redirect, or a «next page» link, to another origin is followed with none of them\n(a CDN serving a file does not need them, and an API that redirects elsewhere must not\nhand them over); an API key in the query is not added there either. Many servers copy\nthe request into the new address (",[48,178,179],{},"return 301 https:\u002F\u002Fother$request_uri","): on any hop\nto another origin, the query pairs named like the API key or holding a secret are\nremoved, and an address that still holds the secret anywhere (its path, encoded or\nnot) is ",[18,182,183],{},"not followed",", with an ",[48,186,187],{},"API:"," error that says why. HTTPS certificates are\nverified. A ",[48,190,191],{},"~\u002F.netrc"," on the worker is never read.",[14,194,195,196,199,200,203],{},"Editing a connection: a new ",[18,197,198],{},"host"," (or a new OAuth2 ",[18,201,202],{},"token URL",", where the client\nsecret goes) needs the secret typed again — a stored secret never follows a connection\nto another server. Another path on the same host keeps it.",[14,205,206,207,210],{},"The secret never appears in a message a person can read: errors, run history (the\nerror and its traceback), the «Try the call» answer, the logs, the audit log and the\nOpenLineage events all show ",[48,208,209],{},"***"," in its place, also when the API echoes it back.",[53,212,214],{"id":213},"datasource-api-call","Datasource «API call»",[58,216,217,226],{},[61,218,219],{},[64,220,221,224],{},[67,222,223],{},"Part",[67,225,72],{},[74,227,228,255,267,277,302,322,334],{},[64,229,230,233],{},[79,231,232],{},"Method, path",[79,234,235,238,239,242,243,246,247,250,251,254],{},[48,236,237],{},"GET"," or ",[48,240,241],{},"POST",", and the path after the base URL (",[48,244,245],{},"\u002Fv1\u002Fquote\u002F{symbol}","). Only a path: the query goes in its own section. Values put in the path by placeholders are URL-encoded (",[48,248,249],{},"A\u002FB"," → ",[48,252,253],{},"A%2FB",").",[64,256,257,260],{},[79,258,259],{},"Query parameters",[79,261,262,263,266],{},"One ",[48,264,265],{},"name=value"," per line. A repeated name is sent twice.",[64,268,269,272],{},[79,270,271],{},"Headers",[79,273,262,274,276],{},[48,275,132],{}," per line; they override the connection's defaults with the same name.",[64,278,279,282],{},[79,280,281],{},"JSON body",[79,283,284,286,287,290,291,294,295,238,298,301],{},[48,285,241],{}," only. Placeholders go ",[18,288,289],{},"inside JSON strings",": ",[48,292,293],{},"{\"since\": \"{last_refresh}\"}",". A string made only of ",[48,296,297],{},"{…:epoch}",[48,299,300],{},"{…:epoch_ms}"," is sent as a number.",[64,303,304,307],{},[79,305,306],{},"Variables",[79,308,309,310,313,314,317,318,35],{},"One per line (up to 10): a list ",[48,311,312],{},"name = value1, value2"," (1000 values each), or a range ",[48,315,316],{},"day = today-7 .. today step 1d"," — see ",[31,319,321],{"href":320},"\u002Fdocs\u002Fplaceholders#variables","placeholders.md",[64,323,324,327],{},[79,325,326],{},"Time zone",[79,328,329,330,333],{},"An IANA name (",[48,331,332],{},"Europe\u002FRome","); it defines «today». Default UTC.",[64,335,336,339],{},[79,337,338],{},"Initial last refresh",[79,340,341,342,345,346,106,349,254],{},"The value of ",[48,343,344],{},"{last_refresh}"," on the first refresh (",[48,347,348],{},"today-365",[48,350,351],{},"2024-01-01",[166,353,355],{"id":354},"dynamic-values","Dynamic values",[14,357,358,359,362],{},"A tiny grammar of our own, parsed and never evaluated — the same language as the file\ndatasources and the Outputs (",[31,360,321],{"href":361},"\u002Fdocs\u002Fplaceholders","):",[58,364,365,375],{},[61,366,367],{},[64,368,369,372],{},[67,370,371],{},"Placeholder",[67,373,374],{},"Value (in the datasource's time zone)",[74,376,377,393,406,423,440,474,499,566,602,615,625,641],{},[64,378,379,387],{},[79,380,381,106,384],{},[48,382,383],{},"{today}",[48,385,386],{},"{yesterday}",[79,388,389,390],{},"a date: ",[48,391,392],{},"2026-10-17",[64,394,395,400],{},[79,396,397],{},[48,398,399],{},"{now}",[79,401,402,403],{},"a date-time: ",[48,404,405],{},"2026-10-17T14:30:05+02:00",[64,407,408,420],{},[79,409,410,106,413,416,417],{},[48,411,412],{},"{start_of_month}",[48,414,415],{},"{start_of_week}"," (Monday), ",[48,418,419],{},"{start_of_year}",[79,421,422],{},"a date",[64,424,425,429],{},[79,426,427],{},[48,428,344],{},[79,430,431,432,435,436,439],{},"when the last ",[18,433,434],{},"successful"," refresh of this datasource ",[18,437,438],{},"started"," (so what changed while it ran is asked again, not lost); on the first refresh, the initial value",[64,441,442,463],{},[79,443,444,447,448,447,451,447,454,447,457,447,460],{},[48,445,446],{},"{today-1}"," ",[48,449,450],{},"{today-7d}",[48,452,453],{},"{today+1d}",[48,455,456],{},"{today-2w}",[48,458,459],{},"{today-1m}",[48,461,462],{},"{today-1y}",[79,464,465,466,469,470,473],{},"offsets: a number alone is days; ",[48,467,468],{},"m"," months, ",[18,471,472],{},"clamped to the end of the month"," (31 March − 1m = 28 February)",[64,475,476,490],{},[79,477,478,447,481,447,484,447,487],{},[48,479,480],{},"{now-2h}",[48,482,483],{},"{now-30min}",[48,485,486],{},"{now+15s}",[48,488,489],{},"{last_refresh-1h}",[79,491,492,493,495,496,498],{},"hours, minutes, seconds: on ",[48,494,399],{}," and ",[48,497,344],{}," only",[64,500,501,515],{},[79,502,503,447,506,447,509,447,512],{},[48,504,505],{},"{today:yyyy-MM-dd}",[48,507,508],{},"{today:ddMMyyyy}",[48,510,511],{},"{today-1:dd\u002FMM\u002Fyy}",[48,513,514],{},"{now:yyyy-MM-dd'T'HH:mm:ss}",[79,516,517,518,447,521,447,524,447,527,447,530,447,533,447,536,539,540,543,544,547,548,550,551,554,555,557,558,561,562,565],{},"a readable format: ",[48,519,520],{},"yyyy",[48,522,523],{},"yy",[48,525,526],{},"MM",[48,528,529],{},"dd",[48,531,532],{},"HH",[48,534,535],{},"mm",[48,537,538],{},"ss",", text in ",[48,541,542],{},"'quotes'"," (",[48,545,546],{},"''"," is an apostrophe). ",[48,549,535],{}," is the minutes right after the hours or right before the seconds, the month otherwise (Excel's rule), so ",[48,552,553],{},"ddmmyyyy"," is a date. ",[48,556,532],{}," is 00-23 and ",[48,559,560],{},"hh"," is refused («use HH»); any other letter is refused («put letters in quotes: 'T'»), and so are a token written twice in a row (",[48,563,564],{},"MMMM",") and a format with no date or time part",[64,567,568,576],{},[79,569,570,447,573],{},[48,571,572],{},"{today:%Y%m%d}",[48,574,575],{},"{now:%Y-%m-%dT%H:%M:%SZ}",[79,577,578,579,582,583,586,587,98,590,495,593,596,597,495,599,601],{},"a strftime format (a ",[48,580,581],{},"%"," outside quotes makes it one). ",[48,584,585],{},"%s"," is refused: use ",[48,588,589],{},":epoch",[48,591,592],{},"%z",[48,594,595],{},"%Z"," only on ",[48,598,399],{},[48,600,344],{}," (on a date they are empty)",[64,603,604,612],{},[79,605,606,447,609],{},[48,607,608],{},"{now:epoch}",[48,610,611],{},"{now:epoch_ms}",[79,613,614],{},"seconds or milliseconds since 1970 (a date is its midnight in the time zone)",[64,616,617,622],{},[79,618,619],{},[48,620,621],{},"{symbol}",[79,623,624],{},"a list variable of the datasource (no offset, no format)",[64,626,627,638],{},[79,628,629,447,632,447,635],{},[48,630,631],{},"{day}",[48,633,634],{},"{day_end}",[48,636,637],{},"{day:yyyyMMdd}",[79,639,640],{},"a range variable and the end of its window (offsets and formats allowed)",[64,642,643,651],{},[79,644,645,447,648],{},[48,646,647],{},"{{",[48,649,650],{},"}}",[79,652,653],{},"a literal brace",[14,655,656,657,659,660,663],{},"An unknown name, a bad offset or format, a variable that is defined but not used, or\n",[48,658,344],{}," without an initial value is refused ",[18,661,662],{},"when the datasource is saved",",\nwith the reason.",[166,665,667],{"id":666},"variables-lists-and-ranges","Variables: lists and ranges",[14,669,670,673,674,677,678,681,682,685],{},[48,671,672],{},"symbol = AAPL, MSFT, ENI.MI"," makes ",[18,675,676],{},"one call per value"," (each with its own\npagination); several variables make every combination. Every row keeps the values in\n",[48,679,680],{},"_param_\u003Cname>"," columns (",[48,683,684],{},"_param_symbol","), the last columns of the table. The number of\ncalls is checked before the first one: over the limit (see below), the datasource is\nrefused.",[14,687,688,689,447,692,694,695,697,698,701,702,705,706,106,709,106,712,106,715,106,718,106,721,106,724,727,728,731,732,35],{},"A ",[18,690,691],{},"range",[48,693,316],{}," makes one call per value too — here the\nseven days before today (the end is excluded) — and ",[48,696,634],{}," is the end of each\nwindow, so an API that takes a period gets consecutive windows that never overlap:\n",[48,699,700],{},"from={day}&to={day_end}",". With append mode keyed on ",[48,703,704],{},"_param_day"," (and the record's own\nkey) this is the incremental import that re-reads the last days every night. Steps\n",[48,707,708],{},"1d",[48,710,711],{},"7d",[48,713,714],{},"1w",[48,716,717],{},"1m",[48,719,720],{},"1y",[48,722,723],{},"1h",[48,725,726],{},"30min","; at most ",[48,729,730],{},"PLACEHOLDERS__MAX_VALUES"," (1000)\nvalues. See ",[31,733,321],{"href":320},[166,735,737],{"id":736},"the-response","The response",[58,739,740,749],{},[61,741,742],{},[64,743,744,747],{},[67,745,746],{},"Option",[67,748,72],{},[74,750,751,770,787,799,826],{},[64,752,753,756],{},[79,754,755],{},"Format",[79,757,758,761,762,765,766,769],{},[18,759,760],{},"JSON"," (default), ",[18,763,764],{},"NDJSON"," (one object per line) or ",[18,767,768],{},"CSV"," (with its separator).",[64,771,772,775],{},[79,773,774],{},"JMESPath",[79,776,777,778,783,784,254],{},"An optional ",[31,779,774],{"href":780,"rel":781},"https:\u002F\u002Fjmespath.org",[782],"nofollow"," expression applied to the whole JSON answer first, for odd shapes (",[48,785,786],{},"results[].quote",[64,788,789,792],{},[79,790,791],{},"Records path",[79,793,794,795,798],{},"A dot path to the records (",[48,796,797],{},"data.items","; a number is a list index). Empty = the answer itself. A path missing from the first page is an error that lists what the answer has; on a later page it means «no more records».",[64,800,801,804],{},[79,802,803],{},"Keyed records",[79,805,806,807,810,811,814,815,818,819,822,823,825],{},"The records are an ",[18,808,809],{},"object whose keys are rows"," — typical of financial time series: ",[48,812,813],{},"{\"2024-01-02\": {\"USD\": 1.1}, \"2024-01-03\": {…}}",". One row per key, the key in ",[48,816,817],{},"_key"," (the first column); a scalar value goes in ",[48,820,821],{},"value",". A record with a field named ",[48,824,817],{}," itself is refused (it would overwrite the key).",[64,827,828,831],{},[79,829,830],{},"List to explode",[79,832,833,834,837,838,841],{},"One field holding a list becomes ",[18,835,836],{},"one row per element"," (an object's fields as ",[48,839,840],{},"field.sub","); an empty list keeps the row with nulls.",[14,843,844,845,848],{},"Flattening: nested objects become columns ",[48,846,847],{},"a.b.c","; lists (other than the exploded\none) become a JSON-text column. Two fields that make the same column name are an error.",[14,850,851,852,855,856,859,860,863,864,867],{},"Types are inferred over all the pages and calls: integers, decimals, booleans, text.\nISO dates ",[18,853,854],{},"stay text"," — cast them in the flow, where the format is yours to say. A\ncolumn that appears on a later page or call is added, NULL before. ",[18,857,858],{},"A column whose\ntypes disagree becomes text"," (an integer and a decimal make a decimal column, not a\nconflict): an API that writes ",[48,861,862],{},"\"N\u002FA\""," in a number field must neither fail the refresh\nnor lose the value. The import reports such columns (",[48,865,866],{},"type_changes",") and the worker\nlogs them.",[166,869,871],{"id":870},"pagination","Pagination",[58,873,874,887],{},[61,875,876],{},[64,877,878,881,884],{},[67,879,880],{},"Type",[67,882,883],{},"How the next page is asked",[67,885,886],{},"When it stops",[74,888,889,899,914,925,940],{},[64,890,891,893,896],{},[79,892,97],{},[79,894,895],{},"one request",[79,897,898],{},"—",[64,900,901,904,911],{},[79,902,903],{},"Page number",[79,905,906,907,910],{},"the page parameter (",[48,908,909],{},"page","), from the first page (default 1)",[79,912,913],{},"on a page with no records",[64,915,916,919,922],{},[79,917,918],{},"Offset and limit",[79,920,921],{},"the offset parameter (from 0, advanced by the rows received) and the limit parameter with the page size",[79,923,924],{},"on a page with no records — not on a short one: an API that caps the page size below the one asked returns short pages that are not the last (one request more, no rows lost)",[64,926,927,930,937],{},[79,928,929],{},"Cursor",[79,931,932,933,936],{},"the cursor parameter with the value at a JSON path of the answer (",[48,934,935],{},"meta.next_cursor","); the first request without it",[79,938,939],{},"when the cursor is missing or empty",[64,941,942,945,956],{},[79,943,944],{},"Next link",[79,946,947,948,951,952,955],{},"the URL in the ",[48,949,950],{},"Link: \u003C…>; rel=\"next\""," header, or at a JSON path (",[48,953,954],{},"links.next","), relative or absolute",[79,957,958],{},"when there is none",[14,960,961,962,965],{},"A cursor or a next link that leads back to a page already read in this call (a self\nlink, two pages pointing at each other) fails the refresh: following it would repeat\nrows. A next link to another host is followed ",[18,963,964],{},"without credentials"," (see above).",[166,967,969],{"id":968},"each-refresh-snapshot-or-append","Each refresh: snapshot or append",[971,972,973,980],"ul",{},[974,975,976,979],"li",{},[18,977,978],{},"Snapshot"," (default): every refresh rebuilds the table from the answers.",[974,981,982,543,985,988,989,992,993,996],{},[18,983,984],{},"Append with key columns",[48,986,987],{},"_key, _param_symbol",", or ",[48,990,991],{},"date, symbol","): the new rows\nare merged into the current snapshot — rows of the snapshot whose key comes again are\n",[18,994,995],{},"replaced",", new keys are added, nothing is duplicated (the same key twice in one\nrefresh: the last one received wins). The types of the snapshot and of the new rows\nare reconciled with the same rule. A refresh that receives no record keeps the table.",[14,998,999],{},"In snapshot mode a refresh that receives no record at all fails and the previous\nsnapshot stays. Every refresh writes a new parquet and the datasource switches to it\nonly on success, as for every database datasource; a refresh in append mode reads the\nsnapshot it extends, and only one refresh of a datasource runs at a time.",[14,1001,1002,1003,35],{},"While it runs, the import says where it is: ",[1004,1005,1006],"em",{},"call 2 of 3 · 7 pages · 700 rows · 12s",[166,1008,1010],{"id":1009},"try-the-call","«Try the call»",[14,1012,1013,1014,1017,1018,1020],{},"Before saving, the datasource form makes ",[18,1015,1016],{},"the first page of the first call"," with\nthe connection's credentials and shows the request as sent (placeholders resolved,\nsecrets as ",[48,1019,209],{}," — in the rows and the excerpt too, when the API echoes a credential,\nJSON-escaped or not), the HTTP status, the number of calls a refresh will make, the next\npage it would ask, an excerpt of the raw answer and the first 20 rows of the table it\nmakes. An answer that cannot become rows (HTTP 404, not JSON, a wrong records path)\nis still shown, with what is wrong. It needs CONNECT on the connection's folder. A\ntrial makes no retries and no wait.",[14,1022,1023],{},"The call of a saved datasource cannot be edited yet: create a new datasource (and\ndelete the old one) to change it.",[53,1025,1027],{"id":1026},"examples-three-free-apis-field-by-field","Examples: three free APIs, field by field",[14,1029,1030],{},"Three public APIs that need no key, each showing one part of the datasource. The field\nnames are the ones the forms show.",[166,1032,1034],{"id":1033},"ecb-exchange-rates-frankfurter-moving-dates-records-keyed-by-date-append","ECB exchange rates (Frankfurter): moving dates, records keyed by date, append",[14,1036,1037,1040],{},[18,1038,1039],{},"Connection"," (type HTTP API)",[58,1042,1043,1052],{},[61,1044,1045],{},[64,1046,1047,1049],{},[67,1048,69],{},[67,1050,1051],{},"Value",[74,1053,1054,1063,1069,1075],{},[64,1055,1056,1058],{},[79,1057,81],{},[79,1059,1060],{},[48,1061,1062],{},"https:\u002F\u002Fapi.frankfurter.dev\u002Fv1",[64,1064,1065,1067],{},[79,1066,92],{},[79,1068,97],{},[64,1070,1071,1073],{},[79,1072,126],{},[79,1074,898],{},[64,1076,1077,1079],{},[79,1078,142],{},[79,1080,1081],{},[48,1082,1083],{},"\u002Flatest",[14,1085,1086],{},[1087,1088],"img",{"alt":1089,"src":1090},"The New connection dialog with type HTTP API: base URL https:\u002F\u002Fapi.frankfurter.dev\u002Fv1, authentication None, test path \u002Flatest, and Connection works","\u002Fmedia\u002Fguide\u002Fconnections\u002Fapi-connection-frankfurter.png",[14,1092,1093],{},[1004,1094,1095,1096,1099,1100,35],{},"The connection after ",[18,1097,1098],{},"Test connection",": no key, the test calls ",[48,1101,1083],{},[14,1103,1104],{},[18,1105,214],{},[58,1107,1108,1116],{},[61,1109,1110],{},[64,1111,1112,1114],{},[67,1113,69],{},[67,1115,1051],{},[74,1117,1118,1127,1137,1150,1159,1165,1174,1187,1194,1202],{},[64,1119,1120,1123],{},[79,1121,1122],{},"Method",[79,1124,1125],{},[48,1126,237],{},[64,1128,1129,1132],{},[79,1130,1131],{},"Path",[79,1133,1134],{},[48,1135,1136],{},"\u002F{today-30}..",[64,1138,1139,1141],{},[79,1140,259],{},[79,1142,1143,495,1146,1149],{},[48,1144,1145],{},"base=EUR",[48,1147,1148],{},"symbols=USD,CHF",", one per line",[64,1151,1152,1154],{},[79,1153,326],{},[79,1155,1156,1158],{},[48,1157,332],{}," (it decides what «today» is)",[64,1160,1161,1163],{},[79,1162,755],{},[79,1164,760],{},[64,1166,1167,1169],{},[79,1168,791],{},[79,1170,1171],{},[48,1172,1173],{},"rates",[64,1175,1176,1179],{},[79,1177,1178],{},"The records are an object keyed by row",[79,1180,1181,1184,1185],{},[18,1182,1183],{},"on",": one row per date, the date in ",[48,1186,817],{},[64,1188,1189,1191],{},[79,1190,871],{},[79,1192,1193],{},"None: one request",[64,1195,1196,1199],{},[79,1197,1198],{},"Each refresh",[79,1200,1201],{},"adds the new rows, replacing those with the same key",[64,1203,1204,1207],{},[79,1205,1206],{},"Key columns",[79,1208,1209],{},[48,1210,817],{},[14,1212,1213,1214,1217,1218,1221,1222,106,1224,106,1227,1230],{},"«Try the call» shows ",[48,1215,1216],{},"GET https:\u002F\u002Fapi.frankfurter.dev\u002Fv1\u002F2026-09-17..?base=EUR&symbols=USD,CHF",",\n",[48,1219,1220],{},"HTTP 200"," and rows with the columns ",[48,1223,817],{},[48,1225,1226],{},"CHF",[48,1228,1229],{},"USD",". The first refresh writes about\n22 rows (business days only); a second one replaces the same dates, so each date stays\nonce.",[14,1232,1233],{},[1087,1234],{"alt":1235,"src":1236},"The API call form of ecb_rates: method GET, path \u002F{today-30}.., the query parameters base and symbols, time zone Europe\u002FRome","\u002Fmedia\u002Fguide\u002Fsources\u002Fapi-call-datasource.png",[14,1238,1239],{},[1004,1240,1241,1242,1245],{},"The datasource ",[48,1243,1244],{},"ecb_rates",": the path with a moving date, the two query parameters, and the time zone that decides what «today» is.",[14,1247,1248],{},[1087,1249],{"alt":1250,"src":1251},"Try the call: the request with its dates resolved, HTTP 200, one call per refresh, 22 rows with the columns _key, CHF and USD","\u002Fmedia\u002Fguide\u002Fsources\u002Fapi-call-try.png",[14,1253,1254],{},[1004,1255,1256,1257,1259,1260,1263,1264,1266,1267,1270],{},"Further down the same form: the records path ",[48,1258,1173],{}," keyed by date, ",[18,1261,1262],{},"append by key"," on ",[48,1265,817],{},", and ",[18,1268,1269],{},"Try the call"," with the request as sent and its first rows. Taken on 28 October 2026, so the path starts on 28 September.",[14,1272,1273,1274,1277,1278,1280,1281,1284],{},"To ask only for what is new, write the path ",[48,1275,1276],{},"\u002F{last_refresh:yyyy-MM-dd}.."," and set\n",[18,1279,338],{}," to ",[48,1282,1283],{},"today-30",": the first refresh reads 30 days, each later one\nstarts where the previous successful refresh started.",[166,1286,1288],{"id":1287},"jsonplaceholder-posts-pagination","JSONPlaceholder posts: pagination",[14,1290,1291,1293,1294,1297,1298,35],{},[18,1292,1039],{},": Base URL ",[48,1295,1296],{},"https:\u002F\u002Fjsonplaceholder.typicode.com",", Authentication None,\nTest path ",[48,1299,1300],{},"\u002Fposts\u002F1",[58,1302,1303,1311],{},[61,1304,1305],{},[64,1306,1307,1309],{},[67,1308,69],{},[67,1310,1051],{},[74,1312,1313,1322,1331,1338,1345,1359],{},[64,1314,1315,1317],{},[79,1316,1131],{},[79,1318,1319],{},[48,1320,1321],{},"\u002Fposts",[64,1323,1324,1326],{},[79,1325,259],{},[79,1327,1328],{},[48,1329,1330],{},"_limit=10",[64,1332,1333,1335],{},[79,1334,791],{},[79,1336,1337],{},"empty (the answer is already a list)",[64,1339,1340,1342],{},[79,1341,1178],{},[79,1343,1344],{},"off",[64,1346,1347,1349],{},[79,1348,871],{},[79,1350,1351,1352,1355,1356],{},"Page number, until an empty page — Page parameter ",[48,1353,1354],{},"_page",", First page ",[48,1357,1358],{},"1",[64,1360,1361,1363],{},[79,1362,1198],{},[79,1364,1365],{},"rebuilds the whole table",[14,1367,1368,1369,106,1372,106,1375,106,1378,1381,1382,1385,1386,1389,1390,1393,1394,1397],{},"100 rows (",[48,1370,1371],{},"userId",[48,1373,1374],{},"id",[48,1376,1377],{},"title",[48,1379,1380],{},"body",") in 10 pages, plus the empty one that ends\nit. The same API also sends a ",[48,1383,1384],{},"Link"," header when the first page is asked for\nexplicitly: add ",[48,1387,1388],{},"_page=1"," to the query parameters and choose ",[18,1391,1392],{},"Next link (Link header or\nthe response)"," with an empty ",[1004,1395,1396],{},"Next URL"," — the same 100 rows in 10 requests, without the\nextra empty one.",[166,1399,1401],{"id":1400},"jsonplaceholder-users-nested-objects","JSONPlaceholder users: nested objects",[14,1403,1404,1405,1408,1409,106,1412,495,1415,35],{},"The same connection, Path ",[48,1406,1407],{},"\u002Fusers",", Pagination None: 10 rows, with the nested objects\nflattened into columns such as ",[48,1410,1411],{},"address.city",[48,1413,1414],{},"address.geo.lat",[48,1416,1417],{},"company.name",[166,1419,1421],{"id":1420},"a-list-variable","A list variable",[14,1423,1424,1425,1428,1429,1431,1432,1435,1436,1438,1439,1441],{},"To read one call per value, add a variable and use it in the call: with\n",[48,1426,1427],{},"symbol = USD, CHF, GBP"," in ",[18,1430,306],{}," and the query parameter ",[48,1433,1434],{},"symbols={symbol}",",\nFrankfurter is called three times and every row says which call it came from in\n",[48,1437,684],{}," (with the key columns ",[48,1440,987],{}," in append mode).",[53,1443,1445],{"id":1444},"limits-and-manners","Limits and manners",[14,1447,1448,1449,1452,1453],{},"Engine settings (backend and workers). A datasource can ",[18,1450,1451],{},"lower"," the pages, calls\nand rows limits for itself and ask for a longer interval between requests, never\nraise a limit above these. ",[18,1454,1455],{},"Going over a limit fails the refresh with a message that\nnames it: the data is never cut silently.",[58,1457,1458,1470],{},[61,1459,1460],{},[64,1461,1462,1465,1468],{},[67,1463,1464],{},"Variable",[67,1466,1467],{},"Default",[67,1469,72],{},[74,1471,1472,1485,1498,1511,1524,1541,1554,1574,1589,1602,1633],{},[64,1473,1474,1479,1482],{},[79,1475,1476],{},[48,1477,1478],{},"API__TIMEOUT_SECONDS",[79,1480,1481],{},"30",[79,1483,1484],{},"Connection and read timeout of one request (the token request too).",[64,1486,1487,1492,1495],{},[79,1488,1489],{},[48,1490,1491],{},"API__MAX_PAGES",[79,1493,1494],{},"1000",[79,1496,1497],{},"Pages of one call: a pagination that never ends stops here.",[64,1499,1500,1505,1508],{},[79,1501,1502],{},[48,1503,1504],{},"API__MAX_CALLS",[79,1506,1507],{},"500",[79,1509,1510],{},"Calls of one refresh (the combinations of the variables), checked before the first.",[64,1512,1513,1518,1521],{},[79,1514,1515],{},[48,1516,1517],{},"API__MAX_ROWS",[79,1519,1520],{},"2000000",[79,1522,1523],{},"Rows received by one refresh, before an append merges them.",[64,1525,1526,1531,1534],{},[79,1527,1528],{},[48,1529,1530],{},"API__MAX_RESPONSE_BYTES",[79,1532,1533],{},"104857600 (100 MB)",[79,1535,1536,1537,1540],{},"One answer, decompressed: it is held in memory to be parsed. A declared ",[48,1538,1539],{},"Content-Length"," over it is refused before reading.",[64,1542,1543,1548,1551],{},[79,1544,1545],{},[48,1546,1547],{},"API__MAX_TOTAL_BYTES",[79,1549,1550],{},"1073741824 (1 GB)",[79,1552,1553],{},"All the answers of one refresh.",[64,1555,1556,1561,1564],{},[79,1557,1558],{},[48,1559,1560],{},"API__MAX_RETRIES",[79,1562,1563],{},"4",[79,1565,1566,1567,1570,1571,1573],{},"Retries of a request that failed with 429, 500, 502, 503, 504 or a network error: after ",[48,1568,1569],{},"Retry-After"," when the API sends it, else 1, 2, 4, 8… seconds. A ",[18,1572,241],{}," is sent again only when it cannot have been acted upon: a 429, or a network error before it was sent (connection refused, connect timeout) — never after a 5xx or a broken answer.",[64,1575,1576,1581,1584],{},[79,1577,1578],{},[48,1579,1580],{},"API__MAX_RETRY_WAIT_SECONDS",[79,1582,1583],{},"120",[79,1585,688,1586,1588],{},[48,1587,1569],{}," longer than this fails the refresh instead of holding a worker.",[64,1590,1591,1596,1599],{},[79,1592,1593],{},[48,1594,1595],{},"API__MAX_REQUEST_SECONDS",[79,1597,1598],{},"300",[79,1600,1601],{},"Wall-clock time of one request, from sending it to the last byte of its answer (the timeout above bounds each wait on the network; this one a server that drips its answer).",[64,1603,1604,1609,1612],{},[79,1605,1606],{},[48,1607,1608],{},"API__MAX_REDIRECTS",[79,1610,1611],{},"5",[79,1613,1614,1615,1618,1619,1622,1623,1626,1627,1629,1630,1632],{},"Redirects followed by one request. A ",[48,1616,1617],{},"303"," (and a ",[48,1620,1621],{},"301","\u002F",[48,1624,1625],{},"302"," after a ",[48,1628,241],{},") continues as ",[48,1631,237],{}," without the body.",[64,1634,1635,1640,1643],{},[79,1636,1637],{},[48,1638,1639],{},"API__MIN_INTERVAL_SECONDS",[79,1641,1642],{},"0",[79,1644,1645],{},"The minimum pause between two requests of one refresh.",[14,1647,1648],{},"The workers make the calls: if your network limits their egress, open it to the APIs.",[14,1650,1651,1652,35],{},"Every message for the user starts with ",[48,1653,187],{},[53,1655,1657],{"id":1656},"lineage-dbt","Lineage, dbt",[14,1659,1660,1661,1664,1665,1668,1669,1672,1673,1676],{},"OpenLineage: a refresh reads one dataset, namespace = the origin of the base URL\n(",[48,1662,1663],{},"https:\u002F\u002Fapi.frankfurter.dev","), name = the base path and the call's path ",[18,1666,1667],{},"as\nwritten",", placeholders included (",[48,1670,1671],{},"\u002Fv1\u002F{today-30}..","), so it is the same dataset at\nevery refresh. Never the query string: it can hold a key. The Lineage page shows the\nconnection feeding the datasource. The dbt export turns an API datasource into a\n",[18,1674,1675],{},"seed"," (like SharePoint and SFTP: dbt has no database to read it from).",[53,1678,1680],{"id":1679},"not-supported-yet","Not supported (yet)",[14,1682,1683],{},"XML answers; OAuth2 flows other than client credentials; secret headers other than\nthe one of the authentication (put a second, non-secret identifier in the default\nheaders); editing the call of a saved datasource; sort keys (the rows keep the order\nof the answers).",[53,1685,1687],{"id":1686},"proofs","Proofs",[14,1689,1690,1693,1694,1697,1698,1701,1702,1705,1706,1709,1710,1713,1714,495,1717,1720,1721,1724,1725,1727,1728,1731,1732,1735],{},[48,1691,1692],{},"backend\u002Ftests\u002Ftest_api_placeholders.py"," (the grammar), ",[48,1695,1696],{},"backend\u002Ftests\u002Ftest_http_api.py","\nand ",[48,1699,1700],{},"test_http_api_route.py"," run against an HTTP server inside the test process\n(",[48,1703,1704],{},"backend\u002Ftests\u002Ffake_api.py","), the gateway's part is in ",[48,1707,1708],{},"gateway\u002Ftests\u002Ftest_api_*.py",".\nAgainst the real dev stack: ",[48,1711,1712],{},"e2e\u002Fapi\u002Ffake_api_server.py"," (a throwaway API served under\ntwo host names, ",[48,1715,1716],{},"zzapi-a",[48,1718,1719],{},"zzapi-b",", to prove that a redirect to another host gets\nno credential), ",[48,1722,1723],{},"e2e\u002Fapi\u002Fapi_proofs.py"," (every authentication, every pagination, 429,\n5xx, redirects, the response cap, list variables, append, placeholders in a time zone,\n",[48,1726,344],{},", no secret anywhere; and the public ",[18,1729,1730],{},"Frankfurter"," API of ECB rates,\nkeyed by date, in append mode) and ",[48,1733,1734],{},"e2e\u002Fapi\u002Fbrowser_forms.py"," (the two forms in\nFirefox, English and Italian).",{"title":1737,"searchDepth":1738,"depth":1739,"links":1740},"",2,3,[1741,1744,1752,1758,1759,1760,1761],{"id":55,"depth":1738,"text":56,"children":1742},[1743],{"id":168,"depth":1739,"text":169},{"id":213,"depth":1738,"text":214,"children":1745},[1746,1747,1748,1749,1750,1751],{"id":354,"depth":1739,"text":355},{"id":666,"depth":1739,"text":667},{"id":736,"depth":1739,"text":737},{"id":870,"depth":1739,"text":871},{"id":968,"depth":1739,"text":969},{"id":1009,"depth":1739,"text":1010},{"id":1026,"depth":1738,"text":1027,"children":1753},[1754,1755,1756,1757],{"id":1033,"depth":1739,"text":1034},{"id":1287,"depth":1739,"text":1288},{"id":1400,"depth":1739,"text":1401},{"id":1420,"depth":1739,"text":1421},{"id":1444,"depth":1738,"text":1445},{"id":1656,"depth":1738,"text":1657},{"id":1679,"depth":1738,"text":1680},{"id":1686,"depth":1738,"text":1687},"An HTTP API connection reaches any REST\u002FJSON API — a financial data broker, a\nCRM, an internal service, a public open-data endpoint. A datasource «API call»\nmakes one or more requests to it at each refresh, turns the records of the answers\ninto one table and keeps it as a snapshot, refreshed by hand or on a schedule like\nevery other database datasource. It is input only: flows read it, nothing is sent\nback to the API. It is created from a folder in Explore with New → Datasource from a\ndatabase, choosing the API connection; the other sources are listed in\nSources.","md",{"source":1765},"docs\u002Fapi-datasource.md",true,"\u002Fdocs\u002Fapi-datasource",{"title":5,"description":1762},"docs\u002Fapi-datasource","DrPG-SmhRAlHtDgufBuBBN33yzBIfGnYNdFjURK96F0",[1772,1791,1846,1875,1907,1936,1962,1988],{"title":1773,"icon":1774,"intro":1775,"items":1776},"Getting started","start","Install it on one machine, learn the few ideas everything else is built on, and run a first flow.",[1777,1779,1782,1785,1788],{"path":1778,"title":1773},"\u002Fdocs\u002Fgetting-started",{"path":1780,"title":1781},"\u002Fdocs\u002Fgetting-started-docker-compose","Walkthrough with Docker Compose",{"path":1783,"title":1784},"\u002Fdocs\u002Fdeploy\u002Fminikube","Try it on your computer (minikube)",{"path":1786,"title":1787},"\u002Fdocs\u002Fconcepts","Concepts",{"path":1789,"title":1790},"\u002Fdocs\u002Foverview","Overview (README)",{"title":1792,"icon":1793,"intro":1794,"items":1795},"Capabilities","capabilities","What you can build: visual flows, analysis on any step, AI questions, pixel-perfect reports, dashboards, Python, R and SQL scripts, containers and the orchestration that ties them together.",[1796,1799,1802,1805,1808,1811,1814,1817,1820,1823,1826,1829,1832,1835,1838,1840,1843],{"path":1797,"title":1798},"\u002Fdocs\u002Fcapabilities","All capabilities",{"path":1800,"title":1801},"\u002Fdocs\u002Fcapabilities\u002Fflows","Low-code flows",{"path":1803,"title":1804},"\u002Fdocs\u002Fcapabilities\u002Fpreview-and-analysis","Preview and analysis",{"path":1806,"title":1807},"\u002Fdocs\u002Fcapabilities\u002Fask-your-data","Ask your data with AI",{"path":1809,"title":1810},"\u002Fdocs\u002Fcapabilities\u002Freports","Pixel-perfect reports",{"path":1812,"title":1813},"\u002Fdocs\u002Fcapabilities\u002Freport-functions","Report functions",{"path":1815,"title":1816},"\u002Fdocs\u002Fcapabilities\u002Fdashboards","Dashboards",{"path":1818,"title":1819},"\u002Fdocs\u002Fscripting","Python and R",{"path":1821,"title":1822},"\u002Fdocs\u002Fsql-scripts","SQL scripts",{"path":1824,"title":1825},"\u002Fdocs\u002Fcapabilities\u002Fnotebooks","marimo notebooks as apps",{"path":1827,"title":1828},"\u002Fdocs\u002Fcontainer-node","Docker images in flows",{"path":1830,"title":1831},"\u002Fdocs\u002Fcapabilities\u002Fflow-groups","Flow groups: orchestration",{"path":1833,"title":1834},"\u002Fdocs\u002Fcapabilities\u002Fscheduling","Scheduling",{"path":1836,"title":1837},"\u002Fdocs\u002Fpushmail","Pushmail: datasources, reports and dashboards by email",{"path":361,"title":1839},"Dynamic dates",{"path":1841,"title":1842},"\u002Fdocs\u002Fcapabilities\u002Fengines","Engines",{"path":1844,"title":1845},"\u002Fdocs\u002Fengines\u002Fengine-differences","Engine differences",{"title":34,"icon":1847,"intro":1848,"items":1849},"sources","Every way data comes in: files, databases, buckets, SFTP folders, HTTP APIs, SharePoint, Google Sheets, and the results of your own flows.",[1850,1852,1855,1858,1861,1864,1866,1869,1872],{"path":33,"title":1851},"All sources",{"path":1853,"title":1854},"\u002Fdocs\u002Fsources\u002Ffiles","Uploaded files",{"path":1856,"title":1857},"\u002Fdocs\u002Fsources\u002Fdatabases","Database tables and queries",{"path":1859,"title":1860},"\u002Fdocs\u002Fs3-datasource","S3 buckets",{"path":1862,"title":1863},"\u002Fdocs\u002Fsftp","SFTP folders",{"path":1767,"title":1865},"HTTP APIs",{"path":1867,"title":1868},"\u002Fdocs\u002Fsources\u002Fsharepoint","Excel on SharePoint",{"path":1870,"title":1871},"\u002Fdocs\u002Fsources\u002Fgoogle-sheets","Google Sheets",{"path":1873,"title":1874},"\u002Fdocs\u002Fsources\u002Fflow-outputs","Results of flows and scripts",{"title":1876,"icon":1877,"intro":1878,"items":1879},"Connections","connections","The systems Tabularia reads from and writes to, how each connection is set up, and who may use it.",[1880,1883,1886,1889,1892,1894,1895,1897,1898,1901,1904],{"path":1881,"title":1882},"\u002Fdocs\u002Fconnections","All connections",{"path":1884,"title":1885},"\u002Fdocs\u002Fconnections\u002Fdatabases","Databases",{"path":1887,"title":1888},"\u002Fdocs\u002Fsqlserver","Microsoft SQL Server driver",{"path":1890,"title":1891},"\u002Fdocs\u002Fconnections\u002Fs3","S3 and compatible storage",{"path":1862,"title":1893},"SFTP servers",{"path":1767,"title":1865},{"path":1867,"title":1896},"SharePoint",{"path":1870,"title":1871},{"path":1899,"title":1900},"\u002Fdocs\u002Fconnections\u002Fsmtp","SMTP email",{"path":1902,"title":1903},"\u002Fdocs\u002Fcontainer-registries","Container registries",{"path":1905,"title":1906},"\u002Fdocs\u002Fconnections\u002Fdestinations","Writing data out",{"title":1908,"icon":1909,"intro":1910,"items":1911},"Integrations","integrations","How Tabularia fits your stack: AI providers, the REST API and webhooks, dbt, OpenLineage, single sign-on, SIEM and monitoring.",[1912,1915,1918,1921,1924,1927,1930,1933],{"path":1913,"title":1914},"\u002Fdocs\u002Fintegrations","All integrations",{"path":1916,"title":1917},"\u002Fdocs\u002Fintegrations\u002Fai","AI: where it is used",{"path":1919,"title":1920},"\u002Fdocs\u002Fapi","API, tokens and webhooks",{"path":1922,"title":1923},"\u002Fdocs\u002Fdbt-export","Export to dbt",{"path":1925,"title":1926},"\u002Fdocs\u002Fopenlineage","OpenLineage",{"path":1928,"title":1929},"\u002Fdocs\u002Fsso-keycloak","Single sign-on with Keycloak",{"path":1931,"title":1932},"\u002Fdocs\u002Fsiem-splunk","Audit log to Splunk",{"path":1934,"title":1935},"\u002Fdocs\u002Fintegrations\u002Fmonitoring","Monitoring",{"title":1937,"icon":1938,"intro":1939,"items":1940},"Security & governance","governance","Who may see and do what, what is recorded, and how data quality and personal data are kept in check.",[1941,1944,1947,1950,1953,1956,1959],{"path":1942,"title":1943},"\u002Fdocs\u002Fgovernance","Overview",{"path":1945,"title":1946},"\u002Fdocs\u002Fgovernance\u002Fpermissions","Permissions",{"path":1948,"title":1949},"\u002Fdocs\u002Fgovernance\u002Faudit-log","Audit log",{"path":1951,"title":1952},"\u002Fdocs\u002Fgovernance\u002Fdata-contracts","Data contracts",{"path":1954,"title":1955},"\u002Fdocs\u002Fdesign\u002Fsso-group-mapping","SSO and group mapping",{"path":1957,"title":1958},"\u002Fdocs\u002Fexamples\u002Fkeycloak","Keycloak example on a laptop",{"path":1960,"title":1961},"\u002Fdocs\u002Fengines\u002Fclickhouse-ai-user","A ClickHouse user for the AI assistant",{"title":1963,"icon":1964,"intro":1965,"items":1966},"Install & operate","operate","Run it in production on Kubernetes, configure it, upgrade it and keep it healthy.",[1967,1970,1973,1976,1979,1982,1985],{"path":1968,"title":1969},"\u002Fdocs\u002Fdeploy\u002Fkubernetes","Kubernetes and Helm",{"path":1971,"title":1972},"\u002Fdocs\u002Fdeploy\u002Fkind","A company-like cluster on one computer (kind)",{"path":1974,"title":1975},"\u002Fdocs\u002Fdeploy\u002Fupgrading","Upgrading, backups and restore",{"path":1977,"title":1978},"\u002Fdocs\u002Fdeploy\u002Frelease-checklist","Install and upgrade checklist",{"path":1980,"title":1981},"\u002Fdocs\u002Fconfiguration","Configuration reference",{"path":1983,"title":1984},"\u002Fdocs\u002Fengine-settings","Settings from the panel",{"path":1986,"title":1987},"\u002Fdocs\u002Fdatabase-migrations","Database migrations",{"title":1989,"icon":1990,"intro":1991,"items":1992},"Reference","reference","Release process and the remaining reference pages.",[1993],{"path":1994,"title":1995},"\u002Fdocs\u002Frelease-process","Release process",1793656371857]