[{"data":1,"prerenderedAt":726},["ShallowReactive",2],{"doc:\u002Fdocs\u002Fs3-datasource":3,"docs-nav":502},{"id":4,"title":5,"body":6,"description":493,"extension":494,"meta":495,"navigation":497,"path":498,"seo":499,"stem":500,"__hash__":501},"docs\u002Fdocs\u002Fs3-datasource.md","S3: the files of a bucket as a datasource",{"type":7,"value":8,"toc":483},"minimark",[9,13,37,52,57,64,239,253,290,297,311,327,336,341,348,436,447,451,466,470],[10,11,5],"h1",{"id":12},"s3-the-files-of-a-bucket-as-a-datasource",[14,15,16,17,21,22,25,26,31,32,36],"p",{},"A datasource ",[18,19,20],"strong",{},"«S3 folder»"," reads the files of a folder of a bucket that match a glob,\nstacks them into one table and keeps it as a snapshot, refreshed by hand or on a\nschedule like every other imported datasource. It reads through an ",[18,23,24],{},"S3 connection"," —\nthe one the S3 Output uses: endpoint (empty = AWS), access key and secret, region,\ndefault bucket — and so reaches any S3-compatible object storage: AWS, MinIO, Cloudflare\nR2, Wasabi, Scaleway, Google Cloud Storage through its S3 API. It is the\n",[27,28,30],"a",{"href":29},"\u002Fdocs\u002Fsftp","SFTP folder"," on object storage; the other sources are listed in\n",[27,33,35],{"href":34},"\u002Fdocs\u002Fsources","Sources",".",[14,38,39,40,43,44,47,48,51],{},"Like the other connections it lives in a folder and needs ",[18,41,42],{},"CONNECT"," on that folder to\nbe used; the secret is encrypted at rest (Fernet), travels encrypted to the worker and\nis decrypted only to sign the requests. The credentials need ",[18,45,46],{},"list"," and ",[18,49,50],{},"read","\nrights on the bucket — nothing else: the datasource never writes or deletes there.",[53,54,56],"h2",{"id":55},"datasource-s3-folder","Datasource «S3 folder»",[14,58,59,60,63],{},"From a folder: ",[18,61,62],{},"New → Datasource from a database",", choose the S3 connection, then:",[65,66,67,80],"table",{},[68,69,70],"thead",{},[71,72,73,77],"tr",{},[74,75,76],"th",{},"Field",[74,78,79],{},"Meaning",[81,82,83,92,117,151,196,204,224],"tbody",{},[71,84,85,89],{},[86,87,88],"td",{},"Bucket",[86,90,91],{},"Empty = the connection's default bucket.",[71,93,94,97],{},[86,95,96],{},"Folder",[86,98,99,100,104,105,108,109,112,113,116],{},"A key prefix taken as a folder: ",[101,102,103],"code",{},"exports\u002Fvendite"," (no leading ",[101,106,107],{},"\u002F","; ",[101,110,111],{},"in"," never takes ",[101,114,115],{},"inbox\u002F","). Empty = the bucket's root.",[71,118,119,122],{},[86,120,121],{},"Files",[86,123,124,125,108,128,47,131,134,135,138,139,142,143,146,147,150],{},"A name or a glob over the keys below the folder: ",[101,126,127],{},"vendite_*.csv",[101,129,130],{},"*",[101,132,133],{},"?"," within a name, ",[101,136,137],{},"**"," for any depth (",[101,140,141],{},"**\u002F*.parquet","). Case-sensitive. ",[101,144,145],{},".."," is refused. ",[18,148,149],{},"Check the files"," lists what it takes.",[71,152,153,156],{},[86,154,155],{},"Format",[86,157,158,162,163,166,167,166,170,173,174,166,177,173,180,166,183,166,186,173,189,166,192,195],{},[159,160,161],"em",{},"From each file's extension"," (",[101,164,165],{},".csv"," ",[101,168,169],{},".txt",[101,171,172],{},".tsv",", ",[101,175,176],{},".parquet",[101,178,179],{},".pq",[101,181,182],{},".xlsx",[101,184,185],{},".xlsm",[101,187,188],{},".xls",[101,190,191],{},".ndjson",[101,193,194],{},".jsonl","), or CSV, Parquet, Excel, NDJSON for every file.",[71,197,198,201],{},[86,199,200],{},"CSV separator, encoding, header; Excel sheet",[86,202,203],{},"As on the SFTP datasource.",[71,205,206,209],{},[86,207,208],{},"Time zone, variables, initial last refresh",[86,210,211,212,216,217,220,221,36],{},"Folder and files may hold ",[27,213,215],{"href":214},"\u002Fdocs\u002Fplaceholders","placeholders"," and variables: ",[101,218,219],{},"daily\u002F{day}\u002Fsales.parquet"," with ",[101,222,223],{},"day = today-7 .. today step 1d",[71,225,226,229],{},[86,227,228],{},"Refresh",[86,230,231,234,235,238],{},[18,232,233],{},"Snapshot",", or ",[18,236,237],{},"append by key",": the rows whose key columns come again replace the old ones.",[14,240,241,242,245,246,249,250,252],{},"The listing starts at the folder plus the glob's fixed leading folders (",[101,243,244],{},"2026\u002F*.csv","\nlists ",[101,247,248],{},"folder\u002F2026\u002F","), so a narrow glob on a big bucket lists little. Keys whose name\nstarts with a dot and «folder» markers (keys ending in ",[101,251,107],{},") are left out.",[14,254,255,256,259,260,263,264,173,267,270,271,274,275,277,278,281,282,285,286,289],{},"Every object that matches is read at ",[18,257,258],{},"every refresh"," and the rows are stacked into\none table, united by column name. Every row says where it comes from: ",[101,261,262],{},"_file"," is the key\nbelow the folder (",[101,265,266],{},"vendite_01.csv",[101,268,269],{},"2026\u002Fq1\u002Fa.csv","; with a dated folder ",[101,272,273],{},"in\u002F{day}",", the\nkey below the fixed part ",[101,276,111],{},"), ",[101,279,280],{},"_file_modified_at"," the object's last modification time\n(UTC) — the two last columns; ",[101,283,284],{},"_param_\u003Cname>"," the values of the variables. The files\nmust be ",[18,287,288],{},"machine readable"," (header in the first row, one name per column, no\nrepeats): anything else is refused, naming the file.",[14,291,292],{},[293,294],"img",{"alt":295,"src":296},"The S3 folder datasource form: name sell_out, connection Partner exchange (S3), bucket empty, folder inbound, files sellout_*.csv, format from each file's extension","\u002Fmedia\u002Fguide\u002Fsources\u002Fs3-folder-datasource.png",[14,298,299],{},[159,300,301,302,306,307,310],{},"The datasource of the ",[27,303,305],{"href":304},"\u002Fdocs\u002Fconnections\u002Fs3#example-a-nightly-exchange-with-a-partner-bucket","S3 connection example",": the distributor's daily files in ",[101,308,309],{},"inbound\u002F",", read from the connection's default bucket.",[14,312,313,314,317,318,321,322,326],{},"With placeholders or variables, a value whose files are missing is ",[18,315,316],{},"skipped and\nlisted"," on the run; no file at all across the values ",[18,319,320],{},"fails"," the refresh (see\n",[27,323,325],{"href":324},"\u002Fdocs\u002Fplaceholders#files-one-expansion-per-value","placeholders.md",").",[14,328,329,332,333,335],{},[18,330,331],{},"Nothing outside the bucket and the folder."," Only the configured bucket is listed,\nonly keys the listing returned below the folder are read, and ",[101,334,145],{}," is refused in the\nfolder, in the glob and in what their placeholders resolve to. No message — errors, the\nrun history, the file check, OpenLineage — holds the secret.",[337,338,340],"h3",{"id":339},"caps","Caps",[14,342,343,344,347],{},"Engine settings (backend and workers), checked on the ",[18,345,346],{},"listing's sizes before\nanything is downloaded",":",[65,349,350,362],{},[68,351,352],{},[71,353,354,357,360],{},[74,355,356],{},"Variable",[74,358,359],{},"Default",[74,361,79],{},[81,363,364,381,394,407,423],{},[71,365,366,371,374],{},[86,367,368],{},[101,369,370],{},"S3_SOURCE__MAX_FILE_BYTES",[86,372,373],{},"536870912 (512 MB)",[86,375,376,377,380],{},"Size of one object (downloaded to the worker's ",[101,378,379],{},"\u002Ftmp",", read in memory). An object that grows past it while being read stops the import.",[71,382,383,388,391],{},[86,384,385],{},[101,386,387],{},"S3_SOURCE__MAX_FILES",[86,389,390],{},"500",[86,392,393],{},"Objects read by one refresh.",[71,395,396,401,404],{},[86,397,398],{},[101,399,400],{},"S3_SOURCE__MAX_TOTAL_BYTES",[86,402,403],{},"2147483648 (2 GB)",[86,405,406],{},"All the matching objects together.",[71,408,409,414,417],{},[86,410,411],{},[101,412,413],{},"S3_SOURCE__MAX_LISTED",[86,415,416],{},"100000",[86,418,419,420,422],{},"Keys the globs of one refresh may scan below their fixed folders, all the values of its variables together: ",[101,421,137],{}," on a huge bucket stops instead of listing it all.",[71,424,425,430,433],{},[86,426,427],{},[101,428,429],{},"S3_SOURCE__MAX_REQUEST_SECONDS",[86,431,432],{},"600",[86,434,435],{},"Wall-clock time of one object's download (each wait on the network is bounded by botocore's 60 s read timeout): a storage that drips the bytes stops the import.",[14,437,438,439,442,443,446],{},"The import reports its progress (file ",[159,440,441],{},"i"," of ",[159,444,445],{},"n",", rows read, saving); the database\ndatasource rules apply (snapshot swap at success, one refresh at a time, schedule,\ndata contract, no sort keys).",[53,448,450],{"id":449},"lineage-dbt","Lineage, dbt",[14,452,453,454,457,458,461,462,465],{},"OpenLineage: an S3 refresh reads ",[101,455,456],{},"s3:\u002F\u002Fbucket"," + ",[101,459,460],{},"\u002Ffolder\u002Fglob"," (the templates as\nwritten); an S3 Output whose key has placeholders is one dataset named after the\ntemplate, the key a run wrote being its version. The dbt export turns an S3 folder\ninto a ",[18,463,464],{},"seed",", like the SFTP and SharePoint ones.",[53,467,469],{"id":468},"proofs","Proofs",[14,471,472,475,476,479,480,36],{},[101,473,474],{},"backend\u002Ftests\u002Ftest_s3_source.py"," (against an in-memory S3 client); the gateway's part\nin ",[101,477,478],{},"gateway\u002Ftests\u002Ftest_placeholders_gateway.py","; against a real MinIO and through the\ndev stack, ",[101,481,482],{},"e2e\u002Fplaceholders\u002F",{"title":484,"searchDepth":485,"depth":486,"links":487},"",2,3,[488,491,492],{"id":55,"depth":485,"text":56,"children":489},[490],{"id":339,"depth":486,"text":340},{"id":449,"depth":485,"text":450},{"id":468,"depth":485,"text":469},"A datasource «S3 folder» reads the files of a folder of a bucket that match a glob,\nstacks them into one table and keeps it as a snapshot, refreshed by hand or on a\nschedule like every other imported datasource. It reads through an S3 connection —\nthe one the S3 Output uses: endpoint (empty = AWS), access key and secret, region,\ndefault bucket — and so reaches any S3-compatible object storage: AWS, MinIO, Cloudflare\nR2, Wasabi, Scaleway, Google Cloud Storage through its S3 API. It is the\nSFTP folder on object storage; the other sources are listed in\nSources.","md",{"source":496},"docs\u002Fs3-datasource.md",true,"\u002Fdocs\u002Fs3-datasource",{"title":5,"description":493},"docs\u002Fs3-datasource","Rn2KKQi-70HJvJBd6Un2__wJPMUGiZnQw7XJsfF6ncY",[503,522,577,605,637,666,692,718],{"title":504,"icon":505,"intro":506,"items":507},"Getting started","start","Install it on one machine, learn the few ideas everything else is built on, and run a first flow.",[508,510,513,516,519],{"path":509,"title":504},"\u002Fdocs\u002Fgetting-started",{"path":511,"title":512},"\u002Fdocs\u002Fgetting-started-docker-compose","Walkthrough with Docker Compose",{"path":514,"title":515},"\u002Fdocs\u002Fdeploy\u002Fminikube","Try it on your computer (minikube)",{"path":517,"title":518},"\u002Fdocs\u002Fconcepts","Concepts",{"path":520,"title":521},"\u002Fdocs\u002Foverview","Overview (README)",{"title":523,"icon":524,"intro":525,"items":526},"Capabilities","capabilities","What you can build: visual flows, analysis on any step, AI questions, pixel-perfect reports, dashboards, Python, R and SQL scripts, containers and the orchestration that ties them together.",[527,530,533,536,539,542,545,548,551,554,557,560,563,566,569,571,574],{"path":528,"title":529},"\u002Fdocs\u002Fcapabilities","All capabilities",{"path":531,"title":532},"\u002Fdocs\u002Fcapabilities\u002Fflows","Low-code flows",{"path":534,"title":535},"\u002Fdocs\u002Fcapabilities\u002Fpreview-and-analysis","Preview and analysis",{"path":537,"title":538},"\u002Fdocs\u002Fcapabilities\u002Fask-your-data","Ask your data with AI",{"path":540,"title":541},"\u002Fdocs\u002Fcapabilities\u002Freports","Pixel-perfect reports",{"path":543,"title":544},"\u002Fdocs\u002Fcapabilities\u002Freport-functions","Report functions",{"path":546,"title":547},"\u002Fdocs\u002Fcapabilities\u002Fdashboards","Dashboards",{"path":549,"title":550},"\u002Fdocs\u002Fscripting","Python and R",{"path":552,"title":553},"\u002Fdocs\u002Fsql-scripts","SQL scripts",{"path":555,"title":556},"\u002Fdocs\u002Fcapabilities\u002Fnotebooks","marimo notebooks as apps",{"path":558,"title":559},"\u002Fdocs\u002Fcontainer-node","Docker images in flows",{"path":561,"title":562},"\u002Fdocs\u002Fcapabilities\u002Fflow-groups","Flow groups: orchestration",{"path":564,"title":565},"\u002Fdocs\u002Fcapabilities\u002Fscheduling","Scheduling",{"path":567,"title":568},"\u002Fdocs\u002Fpushmail","Pushmail: datasources, reports and dashboards by email",{"path":214,"title":570},"Dynamic dates",{"path":572,"title":573},"\u002Fdocs\u002Fcapabilities\u002Fengines","Engines",{"path":575,"title":576},"\u002Fdocs\u002Fengines\u002Fengine-differences","Engine differences",{"title":35,"icon":578,"intro":579,"items":580},"sources","Every way data comes in: files, databases, buckets, SFTP folders, HTTP APIs, SharePoint, Google Sheets, and the results of your own flows.",[581,583,586,589,591,593,596,599,602],{"path":34,"title":582},"All sources",{"path":584,"title":585},"\u002Fdocs\u002Fsources\u002Ffiles","Uploaded files",{"path":587,"title":588},"\u002Fdocs\u002Fsources\u002Fdatabases","Database tables and queries",{"path":498,"title":590},"S3 buckets",{"path":29,"title":592},"SFTP folders",{"path":594,"title":595},"\u002Fdocs\u002Fapi-datasource","HTTP APIs",{"path":597,"title":598},"\u002Fdocs\u002Fsources\u002Fsharepoint","Excel on SharePoint",{"path":600,"title":601},"\u002Fdocs\u002Fsources\u002Fgoogle-sheets","Google Sheets",{"path":603,"title":604},"\u002Fdocs\u002Fsources\u002Fflow-outputs","Results of flows and scripts",{"title":606,"icon":607,"intro":608,"items":609},"Connections","connections","The systems Tabularia reads from and writes to, how each connection is set up, and who may use it.",[610,613,616,619,622,624,625,627,628,631,634],{"path":611,"title":612},"\u002Fdocs\u002Fconnections","All connections",{"path":614,"title":615},"\u002Fdocs\u002Fconnections\u002Fdatabases","Databases",{"path":617,"title":618},"\u002Fdocs\u002Fsqlserver","Microsoft SQL Server driver",{"path":620,"title":621},"\u002Fdocs\u002Fconnections\u002Fs3","S3 and compatible storage",{"path":29,"title":623},"SFTP servers",{"path":594,"title":595},{"path":597,"title":626},"SharePoint",{"path":600,"title":601},{"path":629,"title":630},"\u002Fdocs\u002Fconnections\u002Fsmtp","SMTP email",{"path":632,"title":633},"\u002Fdocs\u002Fcontainer-registries","Container registries",{"path":635,"title":636},"\u002Fdocs\u002Fconnections\u002Fdestinations","Writing data out",{"title":638,"icon":639,"intro":640,"items":641},"Integrations","integrations","How Tabularia fits your stack: AI providers, the REST API and webhooks, dbt, OpenLineage, single sign-on, SIEM and monitoring.",[642,645,648,651,654,657,660,663],{"path":643,"title":644},"\u002Fdocs\u002Fintegrations","All integrations",{"path":646,"title":647},"\u002Fdocs\u002Fintegrations\u002Fai","AI: where it is used",{"path":649,"title":650},"\u002Fdocs\u002Fapi","API, tokens and webhooks",{"path":652,"title":653},"\u002Fdocs\u002Fdbt-export","Export to dbt",{"path":655,"title":656},"\u002Fdocs\u002Fopenlineage","OpenLineage",{"path":658,"title":659},"\u002Fdocs\u002Fsso-keycloak","Single sign-on with Keycloak",{"path":661,"title":662},"\u002Fdocs\u002Fsiem-splunk","Audit log to Splunk",{"path":664,"title":665},"\u002Fdocs\u002Fintegrations\u002Fmonitoring","Monitoring",{"title":667,"icon":668,"intro":669,"items":670},"Security & governance","governance","Who may see and do what, what is recorded, and how data quality and personal data are kept in check.",[671,674,677,680,683,686,689],{"path":672,"title":673},"\u002Fdocs\u002Fgovernance","Overview",{"path":675,"title":676},"\u002Fdocs\u002Fgovernance\u002Fpermissions","Permissions",{"path":678,"title":679},"\u002Fdocs\u002Fgovernance\u002Faudit-log","Audit log",{"path":681,"title":682},"\u002Fdocs\u002Fgovernance\u002Fdata-contracts","Data contracts",{"path":684,"title":685},"\u002Fdocs\u002Fdesign\u002Fsso-group-mapping","SSO and group mapping",{"path":687,"title":688},"\u002Fdocs\u002Fexamples\u002Fkeycloak","Keycloak example on a laptop",{"path":690,"title":691},"\u002Fdocs\u002Fengines\u002Fclickhouse-ai-user","A ClickHouse user for the AI assistant",{"title":693,"icon":694,"intro":695,"items":696},"Install & operate","operate","Run it in production on Kubernetes, configure it, upgrade it and keep it healthy.",[697,700,703,706,709,712,715],{"path":698,"title":699},"\u002Fdocs\u002Fdeploy\u002Fkubernetes","Kubernetes and Helm",{"path":701,"title":702},"\u002Fdocs\u002Fdeploy\u002Fkind","A company-like cluster on one computer (kind)",{"path":704,"title":705},"\u002Fdocs\u002Fdeploy\u002Fupgrading","Upgrading, backups and restore",{"path":707,"title":708},"\u002Fdocs\u002Fdeploy\u002Frelease-checklist","Install and upgrade checklist",{"path":710,"title":711},"\u002Fdocs\u002Fconfiguration","Configuration reference",{"path":713,"title":714},"\u002Fdocs\u002Fengine-settings","Settings from the panel",{"path":716,"title":717},"\u002Fdocs\u002Fdatabase-migrations","Database migrations",{"title":719,"icon":720,"intro":721,"items":722},"Reference","reference","Release process and the remaining reference pages.",[723],{"path":724,"title":725},"\u002Fdocs\u002Frelease-process","Release process",1793656371831]