[{"data":1,"prerenderedAt":1467},["ShallowReactive",2],{"doc:\u002Fdocs\u002Fcapabilities\u002Fflow-groups":3,"docs-nav":1253},{"id":4,"title":5,"body":6,"description":1244,"extension":1245,"meta":1246,"navigation":1248,"path":1249,"seo":1250,"stem":1251,"__hash__":1252},"docs\u002Fdocs\u002Fcapabilities\u002Fflow-groups.md","Flow groups",{"type":7,"value":8,"toc":1229},"minimark",[9,13,22,27,121,125,130,152,159,168,289,292,296,299,324,331,335,338,367,373,377,383,438,445,448,452,455,469,481,485,496,499,567,570,573,577,585,649,1054,1060,1065,1068,1088,1097,1100,1141,1144,1148,1189,1193,1225],[10,11,5],"h1",{"id":12},"flow-groups",[14,15,16,17,21],"p",{},"A flow group runs several pieces of work in a set order, on one schedule: database\nrefreshes, flows, Python or R scripts, images, SQL scripts and emails. It is a grid:\neach row is a ",[18,19,20],"strong",{},"step",", run after the one above it; the cells of a step start together.\nUse one when a report depends on a chain of jobs (import, transform, publish, send)\nand you want the chain to stop as soon as one link breaks, instead of five schedules\nthat run whether or not the data before them is ready.",[23,24,26],"h2",{"id":25},"at-a-glance","At a glance",[28,29,30,41],"table",{},[31,32,33],"thead",{},[34,35,36,39],"tr",{},[37,38],"th",{},[37,40],{},[42,43,44,61,69,77,85,93,101,109],"tbody",{},[34,45,46,50],{},[47,48,49],"td",{},"Where",[47,51,52,53,56,57,60],{},"Explore → ",[18,54,55],{},"New"," → ",[18,58,59],{},"New flow group",", in a folder; the group opens on its own page",[34,62,63,66],{},[47,64,65],{},"What a cell runs",[47,67,68],{},"a flow, the refresh of a datasource, a saved Python or R script, an image (Container), a SQL script that writes a datasource, a Pushmail email",[34,70,71,74],{},[47,72,73],{},"Order",[47,75,76],{},"steps one after the other; the cells of a step in parallel",[34,78,79,82],{},[47,80,81],{},"Size",[47,83,84],{},"at most 50 steps and 20 columns",[34,86,87,90],{},[47,88,89],{},"On failure",[47,91,92],{},"the step finishes, then the group stops: the later steps are skipped",[34,94,95,98],{},[47,96,97],{},"Engines",[47,99,100],{},"the flows run on their production engine, as a scheduled run does",[34,102,103,106],{},[47,104,105],{},"Permissions",[47,107,108],{},"VIEW to open it, EDIT to change it, RUN to run and schedule it, on the group's folder",[34,110,111,114],{},[47,112,113],{},"Rights of a run",[47,115,116,117,120],{},"those of whoever pressed ",[18,118,119],{},"Run now",", or of whoever set the schedule",[23,122,124],{"id":123},"how-it-works","How it works",[126,127,129],"h3",{"id":128},"the-grid","The grid",[14,131,132,133,136,137,140,141,144,145,148,149,151],{},"The page shows the grid on the left and, on the right, ",[18,134,135],{},"Add to the group",": the\nobjects you can put in a cell, with a search by name or folder. Drag one into a cell,\nor choose it from the picker of an empty cell («Drop here or choose»). ",[18,138,139],{},"Add step","\nadds a row, the ",[18,142,143],{},"+"," above the grid adds a column; a step may leave columns empty.\n",[18,146,147],{},"Save"," keeps the group; ",[18,150,119],{}," runs the saved version, so it is greyed out while\nthere are unsaved changes.",[14,153,154],{},[155,156],"img",{"alt":157,"src":158},"The flow group Morning sales: four steps, two refreshes in the first, then a flow, a SQL script and a Pushmail cell, all succeeded; the catalog of objects on the right","\u002Fmedia\u002Fguide\u002Fflow-groups\u002Fmorning-sales-grid.png",[14,160,161],{},[162,163,164,165,167],"em",{},"The group of the example below after a run: every cell green, the outcome and the time of the last run above the grid, ",[18,166,135],{}," on the right.",[28,169,170,183],{},[31,171,172],{},[34,173,174,177,180],{},[37,175,176],{},"Cell",[37,178,179],{},"In the catalog",[37,181,182],{},"What it does",[42,184,185,198,211,230,247,268],{},[34,186,187,190,195],{},[47,188,189],{},"Flow",[47,191,192],{},[18,193,194],{},"Flows",[47,196,197],{},"runs the flow on its production engine; label «Run flow»",[34,199,200,203,208],{},[47,201,202],{},"Refresh",[47,204,205],{},[18,206,207],{},"Datasources to refresh",[47,209,210],{},"imports again a datasource read from a connection: a database table or query, S3, SFTP, an API, SharePoint",[34,212,213,216,221],{},[47,214,215],{},"Saved script",[47,217,218],{},[18,219,220],{},"Saved scripts",[47,222,223,224,229],{},"runs the current version of a saved Python or R script (",[225,226,228],"a",{"href":227},"\u002Fdocs\u002Fscripting#saved-scripts","Python and R","); Kubernetes only",[34,231,232,235,240],{},[47,233,234],{},"Container",[47,236,237],{},[18,238,239],{},"Container (an image)",[47,241,242,243,229],{},"runs an image of a registry granted to you, with no data in and none out (",[225,244,246],{"href":245},"\u002Fdocs\u002Fcontainer-node#in-a-flow-group","the Container node",[34,248,249,252,257],{},[47,250,251],{},"SQL script",[47,253,254],{},[18,255,256],{},"SQL scripts",[47,258,259,260,263,264,267],{},"writes the datasource of a SQL script from its current version, as ",[18,261,262],{},"Refresh now"," on its page (",[225,265,256],{"href":266},"\u002Fdocs\u002Fsql-scripts#in-a-flow-group",")",[34,269,270,273,278],{},[47,271,272],{},"Pushmail",[47,274,275],{},[18,276,277],{},"Pushmail (datasources by email)",[47,279,280,281,285,286,267],{},"sends one email with one ",[282,283,284],"code",{},".xlsx"," file per datasource through an SMTP connection (",[225,287,272],{"href":288},"\u002Fdocs\u002Fpushmail",[14,290,291],{},"A datasource published by a flow cannot be refreshed: put its flow in the group\ninstead. A SQL script goes in a group only once it writes a datasource. The same\nobject twice in one step is refused; the same object in two different steps is\nallowed.",[126,293,295],{"id":294},"a-run","A run",[14,297,298],{},"A group run is queued and picked up by an orchestrator process, like a flow's run.\nFor each step:",[300,301,302,306,309],"ol",{},[303,304,305],"li",{},"Every cell of the step starts. Each cell's work is a run of its own (the flow's,\nthe import's, the script's, the email's), linked to the group's run.",[303,307,308],{},"The step ends when all its cells have ended.",[303,310,311,312,315,316,319,320,323],{},"If any cell failed, the group stops: the cells of the later steps are marked\n",[18,313,314],{},"Skipped"," and the group run ends ",[18,317,318],{},"Failed",", with an error such as\n",[282,321,322],{},"step 2: 1 of 2 cells failed",". Each failed cell keeps its own error.",[14,325,326,327,330],{},"There is no retry at group level. A Container cell has its own ",[18,328,329],{},"Retries","; for the\nothers, run the group again once the cause is fixed.",[126,332,334],{"id":333},"when-the-same-object-is-already-running","When the same object is already running",[14,336,337],{},"A group never starts a second run of something that is already running.",[339,340,341,350,359,364],"ul",{},[303,342,343,345,346,349],{},[18,344,189],{}," and ",[18,347,348],{},"saved script",": the cell waits for the run in progress. That run's\noutcome counts for the cell when it started after the previous step ended (in the\nfirst step, any run counts) and, for a flow, when it runs on the production engine.\nOtherwise it may have read data older than what the earlier steps produced, so the\ncell lets it finish and then runs the object itself. A cell that took over a run\nsays «waited for the run in progress».",[303,351,352,354,355,358],{},[18,353,202],{},": the cell waits for the import already going and takes its outcome.\nThe wait is bounded by ",[282,356,357],{},"ORCHESTRATOR__REFRESH_WAIT_SECONDS"," (600 by default): past\nit the cell fails, even if the import goes on.",[303,360,361,363],{},[18,362,251],{},": as a flow: a write already going counts when it started after the\nprevious step ended.",[303,365,366],{},"While a group runs a flow, the flow's own schedule cannot start it a second time.",[14,368,369,370,372],{},"The group itself runs once at a time: ",[18,371,119],{}," on a group already queued or running\nanswers «The group is already queued or running», and a scheduled slot that comes\nwhile it still runs is skipped.",[126,374,376],{"id":375},"with-whose-rights","With whose rights",[14,378,379,380,382],{},"A run uses the rights of whoever started it: the person who pressed ",[18,381,119],{},", or the\nperson who set the schedule. Every cell checks them again when it runs:",[28,384,385,394],{},[31,386,387],{},[34,388,389,391],{},[37,390,176],{},[37,392,393],{},"Needs",[42,395,396,403,410,417,424,431],{},[34,397,398,400],{},[47,399,189],{},[47,401,402],{},"RUN on the flow's folder, and what the flow itself needs: read access to its sources, EDIT where it publishes, CONNECT on the connections it writes to",[34,404,405,407],{},[47,406,202],{},[47,408,409],{},"RUN on the datasource's folder and CONNECT on its connection",[34,411,412,414],{},[47,413,215],{},[47,415,416],{},"to see the script (VIEW on its folder and the use of its environment), RUN on its folder, read access to its inputs, EDIT where it publishes",[34,418,419,421],{},[47,420,234],{},[47,422,423],{},"the use of the registry and of the image's repository",[34,425,426,428],{},[47,427,251],{},[47,429,430],{},"RUN on its folder, EDIT on the datasource's folder, read access to the datasources it names, CONNECT on its connection",[34,432,433,435],{},[47,434,272],{},[47,436,437],{},"CONNECT on the SMTP connection, read access to every datasource, recipients within the connection's allowed domains",[14,439,440,441,444],{},"A ",[18,442,443],{},"scheduled"," run checks every cell before the first one starts: if the author of the\nschedule lacks anything, nothing runs, and the run is recorded as failed with the list\nof what is missing («Scheduled run not started: the schedule runs with the rights of\n…»). A schedule must not run half a group. A run started by hand checks each cell\nwhen it comes to it.",[14,446,447],{},"Adding a cell also needs those rights, with one more: putting an object in a group\nneeds RUN on its folder, not only VIEW, because the group will run it. A cell already\nin the group stays when someone with fewer rights edits the group.",[126,449,451],{"id":450},"what-switches-the-schedule-off","What switches the schedule off",[14,453,454],{},"The schedule carries its author's authority, so it switches itself off (and the audit\nlog says why) when:",[339,456,457,460,463,466],{},[303,458,459],{},"someone other than its author adds a cell, or changes a Container or Pushmail cell\n(another image, other recipients, other datasources);",[303,461,462],{},"someone other than its author moves the group to another folder;",[303,464,465],{},"a saved script or a SQL script the group runs is changed by someone else so that it\nreads or writes more;",[303,467,468],{},"the author loses RUN on the group's folder, is deactivated or deleted (at the next\nslot).",[14,470,471,472,475,476,480],{},"Removing a cell, or changes by the author, keep it. Set it again from ",[18,473,474],{},"Schedule",",\nas someone who has the rights. See ",[225,477,479],{"href":478},"\u002Fdocs\u002Fcapabilities\u002Fscheduling","Scheduling",".",[126,482,484],{"id":483},"state-halo-and-history","State, halo and history",[14,486,487,488,491,492,495],{},"Above the grid, the run bar shows the run on screen: its state, when it started, how\nlong it took, «started by hand» or «scheduled», its error, and ",[18,489,490],{},"Next run"," when the\ngroup is scheduled. ",[18,493,494],{},"History"," lists the latest 50 runs; choosing one shows its\noutcome on the grid.",[14,497,498],{},"Each cell of the run on screen carries its state in words and a halo of its colour:",[28,500,501,511],{},[31,502,503],{},[34,504,505,508],{},[37,506,507],{},"State",[37,509,510],{},"Means",[42,512,513,521,529,537,544,551,559],{},[34,514,515,518],{},[47,516,517],{},"Waiting",[47,519,520],{},"its step has not started yet, or the orchestrator has no free slot",[34,522,523,526],{},[47,524,525],{},"Running",[47,527,528],{},"the halo breathes while the cell works",[34,530,531,534],{},[47,532,533],{},"Succeeded",[47,535,536],{},"green halo",[34,538,539,541],{},[47,540,318],{},[47,542,543],{},"red halo, with the error under the name",[34,545,546,548],{},[47,547,314],{},[47,549,550],{},"an earlier step failed",[34,552,553,556],{},[47,554,555],{},"Interrupted",[47,557,558],{},"the run ended while the cell was still running (its process died)",[34,560,561,564],{},[47,562,563],{},"Not run",[47,565,566],{},"the run ended (its process died) before the cell's step began",[14,568,569],{},"The page follows a live run by itself and shows a message when it ends («Flow group\n\"…\" completed», or failed with the reason in short).",[14,571,572],{},"A cell whose object sits in a folder you cannot read shows «Not visible to you», and\nits error is hidden from you; a cell whose object was deleted shows «Deleted» and fails\nwhen the group runs.",[23,574,576],{"id":575},"example-the-morning-sales-email","Example: the morning sales email",[14,578,579,580,584],{},"With the sample data (",[225,581,583],{"href":582},"\u002Fdocs\u002Fgetting-started-docker-compose#a-the-sample-data-already-there","the «Sample» folder","):\nevery weekday at 06:30, import the orders again, rebuild the top products, compute the\nrevenue by channel and send both to the sales team.",[28,586,587,600],{},[31,588,589],{},[34,590,591,594,597],{},[37,592,593],{},"Step",[37,595,596],{},"Column 1",[37,598,599],{},"Column 2",[42,601,602,619,629,639],{},[34,603,604,607,614],{},[47,605,606],{},"1",[47,608,609,610,613],{},"Refresh ",[282,611,612],{},"orders"," (Sample ERP)",[47,615,609,616,613],{},[282,617,618],{},"order_rows",[34,620,621,624,627],{},[47,622,623],{},"2",[47,625,626],{},"Flow «Top 20 Products by Revenue»",[47,628],{},[34,630,631,634,637],{},[47,632,633],{},"3",[47,635,636],{},"SQL script «Revenue by channel»",[47,638],{},[34,640,641,644,647],{},[47,642,643],{},"4",[47,645,646],{},"Pushmail «Morning sales report»",[47,648],{},[300,650,651,956,976,987,995,1020,1041],{},[303,652,653,656,657,660,661,664,665,939,942,943,946,947,950,951,953,954,480],{},[18,654,655],{},"The SQL script."," In ",[282,658,659],{},"Sample \u002F Private tables",", ",[18,662,663],{},"New → SQL script",", engine\nDuckDB:",[666,667,672],"pre",{"className":668,"code":669,"language":670,"meta":671,"style":671},"language-sql shiki shiki-themes github-dark-dimmed github-light","SELECT o.canale,\n       EXTRACT(YEAR FROM o.data_ordine) * 100 + EXTRACT(MONTH FROM o.data_ordine) AS anno_mese,\n       round(sum(CAST(r.quantita AS DOUBLE) * CAST(r.prezzo_unitario AS DOUBLE)\n                 * (1 - CAST(r.sconto_pct AS DOUBLE) \u002F 100)), 2) AS ricavo_netto\nFROM \"order_rows\" r\nJOIN \"orders\" o ON r.ordine_id = o.id\nWHERE o.stato \u003C> 'annullato'\nGROUP BY o.canale, anno_mese\nORDER BY anno_mese, ricavo_netto DESC\n","sql","",[282,673,674,696,749,801,847,860,893,912,927],{"__ignoreMap":671},[675,676,679,683,687,690,693],"span",{"class":677,"line":678},"line",1,[675,680,682],{"class":681},"s6PUj","SELECT",[675,684,686],{"class":685},"swcJU"," o",[675,688,480],{"class":689},"ssh_m",[675,691,692],{"class":685},"canale",[675,694,695],{"class":689},",\n",[675,697,699,702,705,708,710,712,715,718,721,724,727,730,733,735,737,739,741,743,746],{"class":677,"line":698},2,[675,700,701],{"class":689},"       EXTRACT(",[675,703,704],{"class":681},"YEAR",[675,706,707],{"class":681}," FROM",[675,709,686],{"class":685},[675,711,480],{"class":689},[675,713,714],{"class":685},"data_ordine",[675,716,717],{"class":689},") ",[675,719,720],{"class":681},"*",[675,722,723],{"class":685}," 100",[675,725,726],{"class":681}," +",[675,728,729],{"class":689}," EXTRACT(",[675,731,732],{"class":681},"MONTH",[675,734,707],{"class":681},[675,736,686],{"class":685},[675,738,480],{"class":689},[675,740,714],{"class":685},[675,742,717],{"class":689},[675,744,745],{"class":681},"AS",[675,747,748],{"class":689}," anno_mese,\n",[675,750,752,755,758,761,763,766,768,771,773,776,779,782,784,787,789,791,793,796,798],{"class":677,"line":751},3,[675,753,754],{"class":685},"       round",[675,756,757],{"class":689},"(",[675,759,760],{"class":685},"sum",[675,762,757],{"class":689},[675,764,765],{"class":685},"CAST",[675,767,757],{"class":689},[675,769,770],{"class":685},"r",[675,772,480],{"class":689},[675,774,775],{"class":685},"quantita",[675,777,778],{"class":681}," AS",[675,780,781],{"class":689}," DOUBLE) ",[675,783,720],{"class":681},[675,785,786],{"class":685}," CAST",[675,788,757],{"class":689},[675,790,770],{"class":685},[675,792,480],{"class":689},[675,794,795],{"class":685},"prezzo_unitario",[675,797,778],{"class":681},[675,799,800],{"class":689}," DOUBLE)\n",[675,802,804,807,810,812,815,817,819,821,823,826,828,830,833,835,838,840,842,844],{"class":677,"line":803},4,[675,805,806],{"class":681},"                 *",[675,808,809],{"class":689}," (",[675,811,606],{"class":685},[675,813,814],{"class":681}," -",[675,816,786],{"class":685},[675,818,757],{"class":689},[675,820,770],{"class":685},[675,822,480],{"class":689},[675,824,825],{"class":685},"sconto_pct",[675,827,778],{"class":681},[675,829,781],{"class":689},[675,831,832],{"class":681},"\u002F",[675,834,723],{"class":685},[675,836,837],{"class":689},")), ",[675,839,623],{"class":685},[675,841,717],{"class":689},[675,843,745],{"class":681},[675,845,846],{"class":689}," ricavo_netto\n",[675,848,850,853,857],{"class":677,"line":849},5,[675,851,852],{"class":681},"FROM",[675,854,856],{"class":855},"sXfbr"," \"order_rows\"",[675,858,859],{"class":689}," r\n",[675,861,863,866,869,872,875,878,880,883,886,888,890],{"class":677,"line":862},6,[675,864,865],{"class":681},"JOIN",[675,867,868],{"class":855}," \"orders\"",[675,870,871],{"class":689}," o ",[675,873,874],{"class":681},"ON",[675,876,877],{"class":685}," r",[675,879,480],{"class":689},[675,881,882],{"class":685},"ordine_id",[675,884,885],{"class":681}," =",[675,887,686],{"class":685},[675,889,480],{"class":689},[675,891,892],{"class":685},"id\n",[675,894,896,899,901,903,906,909],{"class":677,"line":895},7,[675,897,898],{"class":681},"WHERE",[675,900,686],{"class":685},[675,902,480],{"class":689},[675,904,905],{"class":685},"stato",[675,907,908],{"class":681}," \u003C>",[675,910,911],{"class":855}," 'annullato'\n",[675,913,915,918,920,922,924],{"class":677,"line":914},8,[675,916,917],{"class":681},"GROUP BY",[675,919,686],{"class":685},[675,921,480],{"class":689},[675,923,692],{"class":685},[675,925,926],{"class":689},", anno_mese\n",[675,928,930,933,936],{"class":677,"line":929},9,[675,931,932],{"class":681},"ORDER BY",[675,934,935],{"class":689}," anno_mese, ricavo_netto ",[675,937,938],{"class":681},"DESC\n",[940,941],"br",{},"In the ",[18,944,945],{},"Datasource"," card give it the name ",[282,948,949],{},"revenue_by_channel"," and the folder\n",[282,952,659],{},", then ",[18,955,147],{},[303,957,958,656,961,660,964,967,968,345,970,972,973,975],{},[18,959,960],{},"The group.",[282,962,963],{},"Sample \u002F Flows",[18,965,966],{},"New → New flow group",", name it «Morning\nsales». Drag ",[282,969,612],{},[282,971,618],{}," from ",[18,974,207],{}," into the two\ncolumns of step 1: they import at the same time.",[303,977,978,980,981,983,984,986],{},[18,979,139],{},", and drag «Top 20 Products by Revenue» from ",[18,982,194],{}," into step 2. It\nreads ",[282,985,618],{}," only after both imports have succeeded.",[303,988,989,991,992,994],{},[18,990,139],{},", and drag «Revenue by channel» from ",[18,993,256],{}," into step 3.",[303,996,997,999,1000,1002,1003,1006,1007,1010,1011,1013,1014,660,1017,1019],{},[18,998,139],{},", and choose ",[18,1001,277],{}," in step 4: an SMTP\nconnection, To ",[282,1004,1005],{},"sales-team@example.com",", Subject ",[282,1008,1009],{},"Sales {today:yyyy-MM-dd}",", and\nthe datasources «Top 20 Products by Revenue» and ",[282,1012,949],{},", each with\nits file name (",[282,1015,1016],{},"top_products_{today:yyyyMMdd}",[282,1018,949],{},").",[303,1021,1022,953,1024,1026,1027,1029,1030,1032,1036,1038],{},[18,1023,147],{},[18,1025,119],{}," to try it. The cells light up step by step; the email\narrives with two ",[282,1028,284],{}," files.",[940,1031],{},[155,1033],{"alt":1034,"src":1035},"Run now on the flow group: the two refreshes light up together, then the flow, the SQL script and the email, one step after the other","\u002Fmedia\u002Fguide\u002Fflow-groups\u002Frun-a-flow-group.gif",[940,1037],{},[162,1039,1040],{},"Run now: step 1 starts both refreshes at once; each step waits for the one above.",[303,1042,1043,56,1045,1048,1049,1051,1052,480],{},[18,1044,474],{},[282,1046,1047],{},"30 6 * * 1-5"," (Monday to Friday at 06:30, in the installation's\ntime zone) → ",[18,1050,147],{},". The run bar shows ",[18,1053,490],{},[14,1055,1056],{},[155,1057],{"alt":1058,"src":1059},"The schedule dialog of the group with the cron 30 6 * * 1-5 read as At 06:30, Monday through Friday","\u002Fmedia\u002Fguide\u002Fflow-groups\u002Fmorning-sales-schedule.png",[14,1061,1062],{},[162,1063,1064],{},"The group's schedule: the cron, its meaning in words and the time zone it is read in.",[14,1066,1067],{},"What happens when something breaks:",[339,1069,1070,1082,1085],{},[303,1071,1072,1073,1075,1076,1078,1079,1081],{},"The ERP database is down at 06:30: the ",[282,1074,612],{}," refresh fails, ",[282,1077,618],{}," finishes\nits import, then the group stops. Steps 2 to 4 are ",[18,1080,314],{}," and nobody receives\nyesterday's numbers as if they were today's.",[303,1083,1084],{},"Someone started «Top 20 Products by Revenue» by hand at 06:28, before the imports\nended: step 2 waits for that run to end, then runs the flow again on the new data.",[303,1086,1087],{},"The author of the schedule loses CONNECT on «Sample ERP (Postgres)»: the 06:30 run is\nnot started, and its error lists what is missing.",[14,1089,1090,1091,1093,1094,1096],{},"What the schedule's author needs here: RUN on ",[282,1092,963],{}," (the group and the\nflow), RUN on ",[282,1095,659],{}," (the refreshes and the SQL script) and EDIT on\nit (the flow and the script publish there), CONNECT on «Sample ERP (Postgres)» and on\nthe SMTP connection.",[23,1098,105],{"id":1099},"permissions",[28,1101,1102,1112],{},[31,1103,1104],{},[34,1105,1106,1109],{},[37,1107,1108],{},"On the group's folder",[37,1110,1111],{},"You can",[42,1113,1114,1122,1130],{},[34,1115,1116,1119],{},[47,1117,1118],{},"VIEW",[47,1120,1121],{},"open the group, see its grid, its runs and the cells' outcomes (except the objects you cannot read)",[34,1123,1124,1127],{},[47,1125,1126],{},"EDIT",[47,1128,1129],{},"create, change, rename, duplicate, move and delete it",[34,1131,1132,1135],{},[47,1133,1134],{},"RUN",[47,1136,1137,660,1139],{},[18,1138,119],{},[18,1140,474],{},[14,1142,1143],{},"Whoever has no access to the registry or the image of a Container cell cannot open the\ngroup, as for a flow with a Container node. A duplicate is checked with the rights of\nwhoever makes it, and does not take the schedule. Administrators see every group.",[23,1145,1147],{"id":1146},"limits-and-good-to-know","Limits and good to know",[339,1149,1150,1153,1160,1163,1166,1174,1177],{},[303,1151,1152],{},"At most 50 steps and 20 columns. An empty group cannot be run or scheduled.",[303,1154,1155,1156,1159],{},"Cells share the orchestrator's budget, ",[282,1157,1158],{},"ORCHESTRATOR__MAX_CONCURRENT"," (20 by\ndefault per orchestrator process): past it, cells wait. How many really compute at\nonce is then up to the workers and the queue.",[303,1161,1162],{},"A group cannot be deleted while it is queued or running. Deleting it removes its own\nruns; the runs of its cells stay with their flows and datasources.",[303,1164,1165],{},"Flows and datasources can be deleted while a group names them (the cell then shows\n«Deleted»); saved scripts and SQL scripts cannot until they are taken out of the group.",[303,1167,1168,1169,1173],{},"A group's own run sends no webhook event; the flows and refreshes inside it send\ntheirs (",[225,1170,1172],{"href":1171},"\u002Fdocs\u002Fapi#webhooks","webhooks","). The email on failure of a schedule is a\nflow's setting, not a group's.",[303,1175,1176],{},"The grid holds no data between steps: a step hands over through datasources (an\nimport, a flow's Output, a SQL script's datasource), which the next step reads.",[303,1178,1179,1180,1183,1184,1188],{},"The ",[18,1181,1182],{},"Lineage"," page draws a group with what it runs, the images of its Container\ncells and the emails of its Pushmail cells. With ",[225,1185,1187],{"href":1186},"\u002Fdocs\u002Fopenlineage","OpenLineage"," on,\na group's run is the parent of its Container and Pushmail cells and carries what all\nits cells read and wrote; the flows, refreshes and scripts it runs keep their own\njobs and name the group's run.",[23,1190,1192],{"id":1191},"related","Related",[339,1194,1195,1200,1205,1218],{},[303,1196,1197,1199],{},[225,1198,479],{"href":478},": cron, time zone, rights of a schedule",[303,1201,1202],{},[225,1203,194],{"href":1204},"\u002Fdocs\u002Fcapabilities\u002Fflows",[303,1206,1207,660,1209,695,1212,660,1215],{},[225,1208,272],{"href":288},[225,1210,256],{"href":1211},"\u002Fdocs\u002Fsql-scripts",[225,1213,228],{"href":1214},"\u002Fdocs\u002Fscripting",[225,1216,246],{"href":1217},"\u002Fdocs\u002Fcontainer-node",[303,1219,1220,1224],{},[225,1221,1223],{"href":1222},"\u002Fdocs\u002Fplaceholders","Date placeholders",", for the names and subjects of Pushmail",[1226,1227,1228],"style",{},"html pre.shiki code .s6PUj, html code.shiki .s6PUj{--shiki-default:#F47067;--shiki-light:#D73A49}html pre.shiki code .swcJU, html code.shiki .swcJU{--shiki-default:#6CB6FF;--shiki-light:#005CC5}html pre.shiki code .ssh_m, html code.shiki .ssh_m{--shiki-default:#ADBAC7;--shiki-light:#24292E}html pre.shiki code .sXfbr, html code.shiki .sXfbr{--shiki-default:#96D0FF;--shiki-light:#032F62}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}",{"title":671,"searchDepth":698,"depth":751,"links":1230},[1231,1232,1240,1241,1242,1243],{"id":25,"depth":698,"text":26},{"id":123,"depth":698,"text":124,"children":1233},[1234,1235,1236,1237,1238,1239],{"id":128,"depth":751,"text":129},{"id":294,"depth":751,"text":295},{"id":333,"depth":751,"text":334},{"id":375,"depth":751,"text":376},{"id":450,"depth":751,"text":451},{"id":483,"depth":751,"text":484},{"id":575,"depth":698,"text":576},{"id":1099,"depth":698,"text":105},{"id":1146,"depth":698,"text":1147},{"id":1191,"depth":698,"text":1192},"A flow group runs several pieces of work in a set order, on one schedule: database\nrefreshes, flows, Python or R scripts, images, SQL scripts and emails. It is a grid:\neach row is a step, run after the one above it; the cells of a step start together.\nUse one when a report depends on a chain of jobs (import, transform, publish, send)\nand you want the chain to stop as soon as one link breaks, instead of five schedules\nthat run whether or not the data before them is ready.","md",{"source":1247},"docs\u002Fcapabilities\u002Fflow-groups.md",true,"\u002Fdocs\u002Fcapabilities\u002Fflow-groups",{"title":5,"description":1244},"docs\u002Fcapabilities\u002Fflow-groups","m6OyQPcrvz06ZH1bWMbCrkE4GYg7uquvFQmCvI7ISSI",[1254,1273,1317,1349,1381,1408,1433,1459],{"title":1255,"icon":1256,"intro":1257,"items":1258},"Getting started","start","Install it on one machine, learn the few ideas everything else is built on, and run a first flow.",[1259,1261,1264,1267,1270],{"path":1260,"title":1255},"\u002Fdocs\u002Fgetting-started",{"path":1262,"title":1263},"\u002Fdocs\u002Fgetting-started-docker-compose","Walkthrough with Docker Compose",{"path":1265,"title":1266},"\u002Fdocs\u002Fdeploy\u002Fminikube","Try it on your computer (minikube)",{"path":1268,"title":1269},"\u002Fdocs\u002Fconcepts","Concepts",{"path":1271,"title":1272},"\u002Fdocs\u002Foverview","Overview (README)",{"title":1274,"icon":1275,"intro":1276,"items":1277},"Capabilities","capabilities","What you can build: visual flows, analysis on any step, AI questions, pixel-perfect reports, dashboards, Python, R and SQL scripts, containers and the orchestration that ties them together.",[1278,1281,1283,1286,1289,1292,1295,1298,1299,1300,1303,1305,1307,1308,1310,1312,1314],{"path":1279,"title":1280},"\u002Fdocs\u002Fcapabilities","All capabilities",{"path":1204,"title":1282},"Low-code flows",{"path":1284,"title":1285},"\u002Fdocs\u002Fcapabilities\u002Fpreview-and-analysis","Preview and analysis",{"path":1287,"title":1288},"\u002Fdocs\u002Fcapabilities\u002Fask-your-data","Ask your data with AI",{"path":1290,"title":1291},"\u002Fdocs\u002Fcapabilities\u002Freports","Pixel-perfect reports",{"path":1293,"title":1294},"\u002Fdocs\u002Fcapabilities\u002Freport-functions","Report functions",{"path":1296,"title":1297},"\u002Fdocs\u002Fcapabilities\u002Fdashboards","Dashboards",{"path":1214,"title":228},{"path":1211,"title":256},{"path":1301,"title":1302},"\u002Fdocs\u002Fcapabilities\u002Fnotebooks","marimo notebooks as apps",{"path":1217,"title":1304},"Docker images in flows",{"path":1249,"title":1306},"Flow groups: orchestration",{"path":478,"title":479},{"path":288,"title":1309},"Pushmail: datasources, reports and dashboards by email",{"path":1222,"title":1311},"Dynamic dates",{"path":1313,"title":97},"\u002Fdocs\u002Fcapabilities\u002Fengines",{"path":1315,"title":1316},"\u002Fdocs\u002Fengines\u002Fengine-differences","Engine differences",{"title":1318,"icon":1319,"intro":1320,"items":1321},"Sources","sources","Every way data comes in: files, databases, buckets, SFTP folders, HTTP APIs, SharePoint, Google Sheets, and the results of your own flows.",[1322,1325,1328,1331,1334,1337,1340,1343,1346],{"path":1323,"title":1324},"\u002Fdocs\u002Fsources","All sources",{"path":1326,"title":1327},"\u002Fdocs\u002Fsources\u002Ffiles","Uploaded files",{"path":1329,"title":1330},"\u002Fdocs\u002Fsources\u002Fdatabases","Database tables and queries",{"path":1332,"title":1333},"\u002Fdocs\u002Fs3-datasource","S3 buckets",{"path":1335,"title":1336},"\u002Fdocs\u002Fsftp","SFTP folders",{"path":1338,"title":1339},"\u002Fdocs\u002Fapi-datasource","HTTP APIs",{"path":1341,"title":1342},"\u002Fdocs\u002Fsources\u002Fsharepoint","Excel on SharePoint",{"path":1344,"title":1345},"\u002Fdocs\u002Fsources\u002Fgoogle-sheets","Google Sheets",{"path":1347,"title":1348},"\u002Fdocs\u002Fsources\u002Fflow-outputs","Results of flows and scripts",{"title":1350,"icon":1351,"intro":1352,"items":1353},"Connections","connections","The systems Tabularia reads from and writes to, how each connection is set up, and who may use it.",[1354,1357,1360,1363,1366,1368,1369,1371,1372,1375,1378],{"path":1355,"title":1356},"\u002Fdocs\u002Fconnections","All connections",{"path":1358,"title":1359},"\u002Fdocs\u002Fconnections\u002Fdatabases","Databases",{"path":1361,"title":1362},"\u002Fdocs\u002Fsqlserver","Microsoft SQL Server driver",{"path":1364,"title":1365},"\u002Fdocs\u002Fconnections\u002Fs3","S3 and compatible storage",{"path":1335,"title":1367},"SFTP servers",{"path":1338,"title":1339},{"path":1341,"title":1370},"SharePoint",{"path":1344,"title":1345},{"path":1373,"title":1374},"\u002Fdocs\u002Fconnections\u002Fsmtp","SMTP email",{"path":1376,"title":1377},"\u002Fdocs\u002Fcontainer-registries","Container registries",{"path":1379,"title":1380},"\u002Fdocs\u002Fconnections\u002Fdestinations","Writing data out",{"title":1382,"icon":1383,"intro":1384,"items":1385},"Integrations","integrations","How Tabularia fits your stack: AI providers, the REST API and webhooks, dbt, OpenLineage, single sign-on, SIEM and monitoring.",[1386,1389,1392,1395,1398,1399,1402,1405],{"path":1387,"title":1388},"\u002Fdocs\u002Fintegrations","All integrations",{"path":1390,"title":1391},"\u002Fdocs\u002Fintegrations\u002Fai","AI: where it is used",{"path":1393,"title":1394},"\u002Fdocs\u002Fapi","API, tokens and webhooks",{"path":1396,"title":1397},"\u002Fdocs\u002Fdbt-export","Export to dbt",{"path":1186,"title":1187},{"path":1400,"title":1401},"\u002Fdocs\u002Fsso-keycloak","Single sign-on with Keycloak",{"path":1403,"title":1404},"\u002Fdocs\u002Fsiem-splunk","Audit log to Splunk",{"path":1406,"title":1407},"\u002Fdocs\u002Fintegrations\u002Fmonitoring","Monitoring",{"title":1409,"icon":1410,"intro":1411,"items":1412},"Security & governance","governance","Who may see and do what, what is recorded, and how data quality and personal data are kept in check.",[1413,1416,1418,1421,1424,1427,1430],{"path":1414,"title":1415},"\u002Fdocs\u002Fgovernance","Overview",{"path":1417,"title":105},"\u002Fdocs\u002Fgovernance\u002Fpermissions",{"path":1419,"title":1420},"\u002Fdocs\u002Fgovernance\u002Faudit-log","Audit log",{"path":1422,"title":1423},"\u002Fdocs\u002Fgovernance\u002Fdata-contracts","Data contracts",{"path":1425,"title":1426},"\u002Fdocs\u002Fdesign\u002Fsso-group-mapping","SSO and group mapping",{"path":1428,"title":1429},"\u002Fdocs\u002Fexamples\u002Fkeycloak","Keycloak example on a laptop",{"path":1431,"title":1432},"\u002Fdocs\u002Fengines\u002Fclickhouse-ai-user","A ClickHouse user for the AI assistant",{"title":1434,"icon":1435,"intro":1436,"items":1437},"Install & operate","operate","Run it in production on Kubernetes, configure it, upgrade it and keep it healthy.",[1438,1441,1444,1447,1450,1453,1456],{"path":1439,"title":1440},"\u002Fdocs\u002Fdeploy\u002Fkubernetes","Kubernetes and Helm",{"path":1442,"title":1443},"\u002Fdocs\u002Fdeploy\u002Fkind","A company-like cluster on one computer (kind)",{"path":1445,"title":1446},"\u002Fdocs\u002Fdeploy\u002Fupgrading","Upgrading, backups and restore",{"path":1448,"title":1449},"\u002Fdocs\u002Fdeploy\u002Frelease-checklist","Install and upgrade checklist",{"path":1451,"title":1452},"\u002Fdocs\u002Fconfiguration","Configuration reference",{"path":1454,"title":1455},"\u002Fdocs\u002Fengine-settings","Settings from the panel",{"path":1457,"title":1458},"\u002Fdocs\u002Fdatabase-migrations","Database migrations",{"title":1460,"icon":1461,"intro":1462,"items":1463},"Reference","reference","Release process and the remaining reference pages.",[1464],{"path":1465,"title":1466},"\u002Fdocs\u002Frelease-process","Release process",1793656371710]