[{"data":1,"prerenderedAt":3031},["ShallowReactive",2],{"doc:\u002Fdocs\u002Fscripting":3,"docs-nav":2810},{"id":4,"title":5,"body":6,"description":16,"extension":2803,"meta":2804,"navigation":1264,"path":2806,"seo":2807,"stem":2808,"__hash__":2809},"docs\u002Fdocs\u002Fscripting.md","Python and R in flows",{"type":7,"value":8,"toc":2772},"minimark",[9,13,17,67,74,84,91,94,99,176,183,187,190,278,284,318,328,346,349,479,482,487,490,556,559,563,577,588,607,618,626,630,633,637,646,650,653,677,684,688,691,702,705,797,800,880,904,911,936,942,953,959,965,970,974,980,1030,1043,1057,1060,1067,1093,1096,1099,1113,1123,1127,1130,1137,1141,1148,1159,1165,1168,1177,1181,1225,1232,1317,1339,1342,1346,1355,1366,1369,1372,1376,1379,1401,1404,1410,1420,1423,1459,1479,1493,1573,1583,1587,1614,1629,1633,1656,1734,2126,2132,2137,2147,2151,2154,2182,2196,2199,2202,2219,2225,2230,2240,2244,2263,2329,2452,2455,2459,2469,2473,2482,2488,2493,2497,2500,2555,2715,2719,2745,2761,2765,2768],[10,11,5],"h1",{"id":12},"python-and-r-in-flows",[14,15,16],"p",{},"For whoever runs Tabularia on Kubernetes, and whoever writes the scripts.",[18,19,20,28,34,51,57],"ul",{},[21,22,23,27],"li",{},[24,25,26],"strong",{},"Administration → Scripting environments"," defines the Python and R environments\nuser code runs in, outside Tabularia. Each is a language version and a list of\npackages, built once by the cluster into an image that is checked before use. Its\nvariables are what the code finds in its environment.",[21,29,30,33],{},[24,31,32],{},"The Script node"," of the flow editor runs a step's code, one file or several, as a\nKubernetes Job in one of those environments (the design).",[21,35,36,39,40,45,46,50],{},[24,37,38],{},"Saved scripts"," keep code in a folder, with its versions, its inputs and its output,\nrun on their own, on a schedule, in ",[41,42,44],"a",{"href":43},"\u002Fdocs\u002Fcapabilities\u002Fflow-groups","flow groups"," and by Script nodes\n(",[41,47,49],{"href":48},"#saved-scripts","below",").",[21,52,53,56],{},[24,54,55],{},"The Scripting tab"," shows what you may use: your scripts, your environments with\ntheir packages and variables, your container registries.",[21,58,59,62,63,50],{},[24,60,61],{},"Notebooks"," run marimo in the Python environments, as web apps that read the\ndatasources (",[41,64,66],{"href":65},"\u002Fdocs\u002Fcapabilities\u002Fnotebooks","marimo notebooks as apps",[14,68,69],{},[70,71],"img",{"alt":72,"src":73},"The Scripting tab: My scripts lists «Margins under target», in the folder Sales analysis, with its Python environment and its last run, and a New script button; My environments shows the environment Python, Python 3.12, ready, with its installed packages open","\u002Fmedia\u002Fguide\u002Fscripting\u002Fscripting-tab.png",[14,75,76],{},[77,78,79,80,83],"em",{},"The ",[24,81,82],{},"Scripting"," tab, read-only: your saved scripts, and the environments you may write\nfor with what is installed in them. Where Container nodes are on, the registries granted to\nyou are listed too.",[14,85,86,87,90],{},"All of it needs Kubernetes. On Docker Compose, Scripting environments, the palette's\n",[24,88,89],{},"Script"," and the Script nodes of a flow brought from a cluster are greyed out, with\nthe reason.",[14,92,93],{},"The packages come from community indexes only: PyPI for Python, CRAN for R, or your\nmirrors of them.",[95,96,98],"h2",{"id":97},"what-runs-where","What runs where",[100,101,102,117],"table",{},[103,104,105],"thead",{},[106,107,108,111,114],"tr",{},[109,110],"th",{},[109,112,113],{},"where",[109,115,116],{},"what it may do",[118,119,120,135,148,165],"tbody",{},[106,121,122,126,132],{},[123,124,125],"td",{},"the builds",[123,127,128,129],{},"the ",[24,130,131],{},"build namespace",[123,133,134],{},"rootless BuildKit builds the image and pushes it to your registry",[106,136,137,140,145],{},[123,138,139],{},"the checks and the scripts",[123,141,128,142],{},[24,143,144],{},"scripts namespace",[123,146,147],{},"the image runs as user 10001, with a read-only root filesystem, no privilege and no capability",[106,149,150,157,162],{},[123,151,152,153,156],{},"the editor's trials (",[24,154,155],{},"Run",")",[123,158,128,159],{},[24,160,161],{},"trials' namespace",[123,163,164],{},"the same as a script, within a ResourceQuota of its own: trials never take resources from the flows' runs",[106,166,167,170,173],{},[123,168,169],{},"Tabularia",[123,171,172],{},"its own namespace",[123,174,175],{},"the orchestrator builds the images; the gateway and the orchestrator run the other Jobs, read their Pods and logs, and delete them, in the three namespaces above only",[14,177,178,179,182],{},"The scripts never run in Tabularia's namespace and never see its keys. They reach data\nonly through the ",[24,180,181],{},"scripts bucket"," below, with a key the administrators give for it.",[95,184,186],{"id":185},"turning-it-on","Turning it on",[14,188,189],{},"In the chart's values:",[191,192,197],"pre",{"className":193,"code":194,"language":195,"meta":196,"style":196},"language-yaml shiki shiki-themes github-dark-dimmed github-light","scripts:\n  enabled: true\n  registry:\n    repository: registry.example.com\u002Ftabularia   # the images: \u003Crepository>\u002Ftabularia-env-\u003Cid>:v\u003Cn>\n    pushSecret: registry-push    # dockerconfigjson Secret in the build namespace\n    pullSecret: registry-pull    # dockerconfigjson Secret in the scripts namespace\n","yaml","",[198,199,200,213,226,234,250,264],"code",{"__ignoreMap":196},[201,202,205,209],"span",{"class":203,"line":204},"line",1,[201,206,208],{"class":207},"shb1k","scripts",[201,210,212],{"class":211},"ssh_m",":\n",[201,214,216,219,222],{"class":203,"line":215},2,[201,217,218],{"class":207},"  enabled",[201,220,221],{"class":211},": ",[201,223,225],{"class":224},"swcJU","true\n",[201,227,229,232],{"class":203,"line":228},3,[201,230,231],{"class":207},"  registry",[201,233,212],{"class":211},[201,235,237,240,242,246],{"class":203,"line":236},4,[201,238,239],{"class":207},"    repository",[201,241,221],{"class":211},[201,243,245],{"class":244},"sXfbr","registry.example.com\u002Ftabularia",[201,247,249],{"class":248},"sgHix","   # the images: \u003Crepository>\u002Ftabularia-env-\u003Cid>:v\u003Cn>\n",[201,251,253,256,258,261],{"class":203,"line":252},5,[201,254,255],{"class":207},"    pushSecret",[201,257,221],{"class":211},[201,259,260],{"class":244},"registry-push",[201,262,263],{"class":248},"    # dockerconfigjson Secret in the build namespace\n",[201,265,267,270,272,275],{"class":203,"line":266},6,[201,268,269],{"class":207},"    pullSecret",[201,271,221],{"class":211},[201,273,274],{"class":244},"registry-pull",[201,276,277],{"class":248},"    # dockerconfigjson Secret in the scripts namespace\n",[14,279,280,283],{},[198,281,282],{},"helm upgrade"," creates three namespaces next to the release's:",[18,285,286,296,309],{},[21,287,288,291,292,295],{},[198,289,290],{},"\u003Crelease namespace>-scripts",", at the ",[198,293,294],{},"restricted"," Pod Security level;",[21,297,298,301,302,304,305,308],{},[198,299,300],{},"\u003Crelease namespace>-scripts-dev",", ",[198,303,294],{}," too, for the editor's trials, with a\nResourceQuota (",[198,306,307],{},"scripts.trialQuota",": 4 CPUs and 8 GiB requested, 8 CPUs and 16 GiB at\nmost, 10 pods, all trials together). A trial that does not fit fails at once and\nsays the quota is full;",[21,310,311,291,314,317],{},[198,312,313],{},"\u003Crelease namespace>-script-builds",[198,315,316],{},"privileged"," level.",[14,319,320,321,324,325,327],{},"Rootless BuildKit needs a user namespace of its own, which needs seccomp and AppArmor\n",[198,322,323],{},"Unconfined",", and only ",[198,326,316],{}," admits that. Its pods still run as a user and are\nnot privileged containers.",[14,329,330,331,334,335,301,338,341,342,345],{},"To manage the namespaces yourself, set ",[198,332,333],{},"scripts.createNamespaces: false"," and name them\nwith ",[198,336,337],{},"scripts.namespace",[198,339,340],{},"scripts.trialNamespace"," and ",[198,343,344],{},"scripts.buildNamespace",".",[14,347,348],{},"The Secrets hold the registry's credentials: the push one in the builds' namespace, the\npull one in the scripts' and in the trials' namespaces:",[191,350,354],{"className":351,"code":352,"language":353,"meta":196,"style":196},"language-bash shiki shiki-themes github-dark-dimmed github-light","kubectl -n \u003Cnamespace>-script-builds create secret docker-registry registry-push \\\n  --docker-server=registry.example.com --docker-username=… --docker-password=…\nkubectl -n \u003Cnamespace>-scripts create secret docker-registry registry-pull \\\n  --docker-server=registry.example.com --docker-username=… --docker-password=…\nkubectl -n \u003Cnamespace>-scripts-dev create secret docker-registry registry-pull \\\n  --docker-server=registry.example.com --docker-username=… --docker-password=…\n","bash",[198,355,356,397,408,436,444,471],{"__ignoreMap":196},[201,357,358,362,365,369,372,375,378,381,384,387,390,393],{"class":203,"line":204},[201,359,361],{"class":360},"sqRhv","kubectl",[201,363,364],{"class":224}," -n",[201,366,368],{"class":367},"s6PUj"," \u003C",[201,370,371],{"class":244},"namespac",[201,373,374],{"class":211},"e",[201,376,377],{"class":367},">",[201,379,380],{"class":244},"-script-builds",[201,382,383],{"class":244}," create",[201,385,386],{"class":244}," secret",[201,388,389],{"class":244}," docker-registry",[201,391,392],{"class":244}," registry-push",[201,394,396],{"class":395},"sxsTv"," \\\n",[201,398,399,402,405],{"class":203,"line":215},[201,400,401],{"class":224},"  --docker-server=registry.example.com",[201,403,404],{"class":224}," --docker-username=…",[201,406,407],{"class":224}," --docker-password=…\n",[201,409,410,412,414,416,418,420,422,425,427,429,431,434],{"class":203,"line":228},[201,411,361],{"class":360},[201,413,364],{"class":224},[201,415,368],{"class":367},[201,417,371],{"class":244},[201,419,374],{"class":211},[201,421,377],{"class":367},[201,423,424],{"class":244},"-scripts",[201,426,383],{"class":244},[201,428,386],{"class":244},[201,430,389],{"class":244},[201,432,433],{"class":244}," registry-pull",[201,435,396],{"class":395},[201,437,438,440,442],{"class":203,"line":236},[201,439,401],{"class":224},[201,441,404],{"class":224},[201,443,407],{"class":224},[201,445,446,448,450,452,454,456,458,461,463,465,467,469],{"class":203,"line":252},[201,447,361],{"class":360},[201,449,364],{"class":224},[201,451,368],{"class":367},[201,453,371],{"class":244},[201,455,374],{"class":211},[201,457,377],{"class":367},[201,459,460],{"class":244},"-scripts-dev",[201,462,383],{"class":244},[201,464,386],{"class":244},[201,466,389],{"class":244},[201,468,433],{"class":244},[201,470,396],{"class":395},[201,472,473,475,477],{"class":203,"line":266},[201,474,401],{"class":224},[201,476,404],{"class":224},[201,478,407],{"class":224},[14,480,481],{},"Give the push account write access to the repository path, and the pull account read\naccess only. A registry that takes anonymous pulls needs no pull secret.",[483,484,486],"h3",{"id":485},"a-registry-with-your-companys-certificate","A registry with your company's certificate",[14,488,489],{},"Two programs talk to the registry, and both must trust its certificate:",[18,491,492,546],{},[21,493,494,496,497,539,542,545],{},[24,495,125],{}," push the images. When the certificate is signed by your company's\nauthority, give the chart that authority's certificate:",[191,498,500],{"className":193,"code":499,"language":195,"meta":196,"style":196},"scripts:\n  registry:\n    caCert: |\n      -----BEGIN CERTIFICATE-----\n      …\n      -----END CERTIFICATE-----\n",[198,501,502,508,514,524,529,534],{"__ignoreMap":196},[201,503,504,506],{"class":203,"line":204},[201,505,208],{"class":207},[201,507,212],{"class":211},[201,509,510,512],{"class":203,"line":215},[201,511,231],{"class":207},[201,513,212],{"class":211},[201,515,516,519,521],{"class":203,"line":228},[201,517,518],{"class":207},"    caCert",[201,520,221],{"class":211},[201,522,523],{"class":367},"|\n",[201,525,526],{"class":203,"line":236},[201,527,528],{"class":244},"      -----BEGIN CERTIFICATE-----\n",[201,530,531],{"class":203,"line":252},[201,532,533],{"class":244},"      …\n",[201,535,536],{"class":203,"line":266},[201,537,538],{"class":244},"      -----END CERTIFICATE-----\n",[540,541],"br",{},[198,543,544],{},"scripts.registry.insecure: true"," instead skips the verification, or allows plain\nHTTP: for a test registry only.",[21,547,548,551,552,555],{},[24,549,550],{},"the nodes"," pull the images, as for any pod. Their container runtime must trust\nthe authority too: install it on the nodes the way your platform does it (a node\nimage, a DaemonSet that writes it to containerd's ",[198,553,554],{},"certs.d","). A registry in plain HTTP\nmust likewise be allowed on the nodes. Tabularia cannot do this from a pod.",[14,557,558],{},"When either side refuses the registry, the environment's version fails at once and\nsays which side and why. So does a run whose image a node cannot download: the\ncredentials are refused, the nodes do not trust the certificate, the image is no\nlonger there. It does not wait for its timeout.",[483,560,562],{"id":561},"network-policies","Network policies",[14,564,565,566,569,570,573,574,345],{},"With ",[198,567,568],{},"networkPolicy.egressCIDRs"," set, the engine and the workers reach only the\naddresses listed. The engine also writes each Script node's input to the scripts\nbucket: include its address. A bucket, broker or ClickHouse that is a Service of the\ncluster, in another namespace (",[198,571,572],{},"minio.storage.svc.cluster.local","), is allowed from its\nhost name; for other namespaces, list them in ",[198,575,576],{},"networkPolicy.egressNamespaces",[14,578,579,580,583,584,587],{},"The scripts themselves, their trials and the image builds get a policy of their own\n(",[198,581,582],{},"scripts.blockTabularia",", on by default): they reach the internet and the other\nnamespaces of the cluster (the scripts bucket, your services), but ",[24,585,586],{},"never the pods of\nTabularia's namespace"," (the gateway, the engine, the workers) nor the cloud metadata\naddresses, and nothing reaches into them. Outside the cluster the private ranges are\nclosed too (10\u002F8, 172.16\u002F12, 192.168\u002F16, 100.64\u002F10, and fc00::\u002F7 on IPv6): on most\nnetwork plugins they also hold the cluster's pods.",[14,589,590,591,594,595,598,599,602,603,606],{},"What the scripts must reach on a private address goes in\n",[198,592,593],{},"networkPolicy.userCodePrivateCIDRs",": a company network, a database on ",[198,596,597],{},"10.20.0.0\u002F16","\nfor example, and above all ",[24,600,601],{},"the registry the builds push to"," when it has a private\naddress, as company registries usually do (the nodes pull the images, so pulls do not\ndepend on it), and a scripts bucket on a private endpoint. Never list a range that\nholds the cluster's pods or Services, nor Tabularia's database, broker or storage. A\nnode-local DNS cache on a link-local address (",[198,604,605],{},"169.254.20.10\u002F32",") is listed the same way.",[14,608,609,610,613,614,617],{},"Tabularia's database, broker and storage are out of the scripts' reach when they are\noutside the cluster on a private address that is not listed, or in the release's\nnamespace. In ",[24,611,612],{},"another namespace of the cluster"," the scripts can reach them: give\nthat namespace a policy that lets in Tabularia's namespace only, as\n",[198,615,616],{},"infrastructure\u002Fminikube\u002Fdependencies.yaml"," does for its database and queue.",[14,619,620,621,625],{},"Without a network plugin that enforces network policies the policy is written but has\nno effect. Some enforce only what comes into a pod: kind's own plugin (kindnet) blocks\ntraffic into the dependencies above, but not what the scripts send out, which is why\n",[41,622,624],{"href":623},"\u002Fdocs\u002Fdeploy\u002Fkind","the kind guide"," installs Calico. Calico and Cilium enforce both.",[483,627,629],{"id":628},"nodes-of-more-than-one-architecture","Nodes of more than one architecture",[14,631,632],{},"A build runs on one node and the image is built for that node's architecture (amd64,\narm64). Tabularia records it, and sends the check of the image and every run of its\nscripts to nodes of the same architecture. On a cluster with amd64 and arm64 nodes,\nthe environment works on the nodes of the architecture its build landed on.",[483,634,636],{"id":635},"policies-on-pods","Policies on pods",[14,638,639,640,642,643,645],{},"The builds need the ",[198,641,316],{}," Pod Security level for their namespace and\nseccomp and AppArmor ",[198,644,323],{}," (see above). A cluster whose admission policies\n(Kyverno, Gatekeeper) forbid that for every namespace cannot build environments.",[483,647,649],{"id":648},"a-cluster-without-internet-access","A cluster without internet access",[14,651,652],{},"The builds download the base image, BuildKit and the packages. Point all four at your\nmirrors:",[18,654,655,661,667,672],{},[21,656,657,660],{},[198,658,659],{},"scripts.builderImage",";",[21,662,663,666],{},[198,664,665],{},"scripts.baseImages",": the language versions offered and their images, which can\nalso be pinned by digest here;",[21,668,669,660],{},[198,670,671],{},"scripts.pypiIndexUrl",[21,673,674,345],{},[198,675,676],{},"scripts.cranUrl",[14,678,679,680,345],{},"Every variable is in the ",[41,681,683],{"href":682},"\u002Fdocs\u002Fconfiguration#scripting-environments-kubernetes","configuration reference",[95,685,687],{"id":686},"the-scripts-bucket","The scripts bucket",[14,689,690],{},"Create a bucket for the scripts only, and a key that works on that bucket and nothing\nelse. Tabularia writes the input of each Script node there and deletes it when the run\nends; the scripts read it, and write their output, with the same key. Tabularia's own\nstorage never reaches a script.",[14,692,693,694,697,698,701],{},"Give the key the right to ",[24,695,696],{},"read, write and delete objects",", and ",[24,699,700],{},"not"," the right to\nlist the bucket. The scripts do not need to list it: they read and write files whose\nnames they are given. Without that right, a script cannot find the files of another\nrun in progress either.",[14,703,704],{},"AWS S3, an IAM user or role with:",[191,706,710],{"className":707,"code":708,"language":709,"meta":196,"style":196},"language-json shiki shiki-themes github-dark-dimmed github-light","{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [{\n    \"Effect\": \"Allow\",\n    \"Action\": [\"s3:GetObject\", \"s3:PutObject\", \"s3:DeleteObject\"],\n    \"Resource\": \"arn:aws:s3:::tabularia-scripts\u002F*\"\n  }]\n}\n","json",[198,711,712,717,731,739,751,775,785,791],{"__ignoreMap":196},[201,713,714],{"class":203,"line":204},[201,715,716],{"class":211},"{\n",[201,718,719,723,725,728],{"class":203,"line":215},[201,720,722],{"class":721},"sJK54","  \"Version\"",[201,724,221],{"class":211},[201,726,727],{"class":244},"\"2012-10-17\"",[201,729,730],{"class":211},",\n",[201,732,733,736],{"class":203,"line":228},[201,734,735],{"class":721},"  \"Statement\"",[201,737,738],{"class":211},": [{\n",[201,740,741,744,746,749],{"class":203,"line":236},[201,742,743],{"class":721},"    \"Effect\"",[201,745,221],{"class":211},[201,747,748],{"class":244},"\"Allow\"",[201,750,730],{"class":211},[201,752,753,756,759,762,764,767,769,772],{"class":203,"line":252},[201,754,755],{"class":721},"    \"Action\"",[201,757,758],{"class":211},": [",[201,760,761],{"class":244},"\"s3:GetObject\"",[201,763,301],{"class":211},[201,765,766],{"class":244},"\"s3:PutObject\"",[201,768,301],{"class":211},[201,770,771],{"class":244},"\"s3:DeleteObject\"",[201,773,774],{"class":211},"],\n",[201,776,777,780,782],{"class":203,"line":266},[201,778,779],{"class":721},"    \"Resource\"",[201,781,221],{"class":211},[201,783,784],{"class":244},"\"arn:aws:s3:::tabularia-scripts\u002F*\"\n",[201,786,788],{"class":203,"line":787},7,[201,789,790],{"class":211},"  }]\n",[201,792,794],{"class":203,"line":793},8,[201,795,796],{"class":211},"}\n",[14,798,799],{},"MinIO, the same policy:",[191,801,803],{"className":351,"code":802,"language":353,"meta":196,"style":196},"mc mb local\u002Ftabularia-scripts\nmc admin policy create local tabularia-scripts policy.json   # the JSON above\nmc admin user add local tabularia-scripts '\u003Csecret key>'\nmc admin policy attach local tabularia-scripts --user tabularia-scripts\n",[198,804,805,816,840,859],{"__ignoreMap":196},[201,806,807,810,813],{"class":203,"line":204},[201,808,809],{"class":360},"mc",[201,811,812],{"class":244}," mb",[201,814,815],{"class":244}," local\u002Ftabularia-scripts\n",[201,817,818,820,823,826,828,831,834,837],{"class":203,"line":215},[201,819,809],{"class":360},[201,821,822],{"class":244}," admin",[201,824,825],{"class":244}," policy",[201,827,383],{"class":244},[201,829,830],{"class":244}," local",[201,832,833],{"class":244}," tabularia-scripts",[201,835,836],{"class":244}," policy.json",[201,838,839],{"class":248},"   # the JSON above\n",[201,841,842,844,846,849,852,854,856],{"class":203,"line":228},[201,843,809],{"class":360},[201,845,822],{"class":244},[201,847,848],{"class":244}," user",[201,850,851],{"class":244}," add",[201,853,830],{"class":244},[201,855,833],{"class":244},[201,857,858],{"class":244}," '\u003Csecret key>'\n",[201,860,861,863,865,867,870,872,874,877],{"class":203,"line":236},[201,862,809],{"class":360},[201,864,822],{"class":244},[201,866,825],{"class":244},[201,868,869],{"class":244}," attach",[201,871,830],{"class":244},[201,873,833],{"class":244},[201,875,876],{"class":224}," --user",[201,878,879],{"class":244}," tabularia-scripts\n",[14,881,882,883,886,887,890,891,894,895,730,898,341,901,345],{},"On ",[24,884,885],{},"Scaleway",", use an IAM application with a bucket policy that grants it those three\nactions on ",[198,888,889],{},"tabularia-scripts\u002F*",". On ",[24,892,893],{},"Google Cloud Storage",", use an HMAC key of a\nservice account whose role on the bucket holds only ",[198,896,897],{},"storage.objects.get",[198,899,900],{},"storage.objects.create",[198,902,903],{},"storage.objects.delete",[14,905,906,907,910],{},"Add a ",[24,908,909],{},"lifecycle rule"," that expires the bucket's objects after a day. It removes what\na run that crashed left behind:",[191,912,914],{"className":351,"code":913,"language":353,"meta":196,"style":196},"mc ilm rule add --expire-days 1 local\u002Ftabularia-scripts\n",[198,915,916],{"__ignoreMap":196},[201,917,918,920,923,926,928,931,934],{"class":203,"line":204},[201,919,809],{"class":360},[201,921,922],{"class":244}," ilm",[201,924,925],{"class":244}," rule",[201,927,851],{"class":244},[201,929,930],{"class":224}," --expire-days",[201,932,933],{"class":224}," 1",[201,935,815],{"class":244},[14,937,938,939,941],{},"In ",[24,940,26],{},", fill in:",[18,943,944,947,950],{},[21,945,946],{},"the endpoint (empty for AWS);",[21,948,949],{},"the bucket and the region;",[21,951,952],{},"the access key and the secret key.",[14,954,955,958],{},[24,956,957],{},"Test"," writes, reads and deletes a probe object from the engine, as the runs will.\nIt also warns when the key can list the bucket. The secret key is encrypted and cannot\nbe read back.",[14,960,961],{},[70,962],{"alt":963,"src":964},"Administration, Scripting environments: the bucket for the scripts, with the endpoint of a MinIO service in the cluster, the bucket tabularia-scripts, the region us-east-1, the access key scripts and the secret key shown only as set, Save and Test; below, the environment Python 3.12 with version 1 in use","\u002Fmedia\u002Fguide\u002Fscripting\u002Fadmin-scripting-environments.png",[14,966,967],{},[77,968,969],{},"The scripts bucket and the environments, in Administration. The secret key is never shown\nagain: the field only says it is set.",[95,971,973],{"id":972},"environments","Environments",[14,975,976,979],{},[24,977,978],{},"New environment"," asks for a name, a language and its version, and the packages.\nPackages go one per line:",[18,981,982,1010],{},[21,983,984,987,988,221,991,994,995,301,998,1001,1002,1005,1006,1009],{},[24,985,986],{},"Python",", as in ",[198,989,990],{},"requirements.txt",[198,992,993],{},"scikit-learn==1.5.2",". ",[198,996,997],{},"polars",[198,999,1000],{},"pyarrow"," and\n",[198,1003,1004],{},"marimo"," (for the ",[41,1007,1008],{"href":65},"notebooks",") are always installed.",[21,1011,1012,221,1015,1018,1019,1022,1023,1026,1027,1029],{},[24,1013,1014],{},"R",[198,1016,1017],{},"dplyr",", or ",[198,1020,1021],{},"data.table@1.16.0"," for a fixed version. ",[198,1024,1025],{},"arrow",", with S3 support,\nis always installed. CRAN publishes sources, so R packages compile during the build:\nthe images carry the system libraries the usual packages need, and a build with\n",[198,1028,1025],{}," takes several minutes. It happens once per version.",[14,1031,1032,1033,341,1036,1039,1040,1042],{},"A build gets its own CPU and memory, ",[198,1034,1035],{},"SCRIPTS__BUILD_CPU",[198,1037,1038],{},"SCRIPTS__BUILD_MEMORY","\n(2 cores and 4 GiB by default). The memory is requested whole, so a build lands only on\na node that has it. It runs one compiler per 1.5 GiB, at most one per core: a C++\ncompiler of ",[198,1041,1025],{}," takes over 1 GB. A build that runs out of memory fails saying so;\non large nodes, raising both makes R builds faster.",[14,1044,1045,1048,1049,1052,1053,1056],{},[24,1046,1047],{},"Search PyPI"," \u002F ",[24,1050,1051],{},"Search CRAN"," above the list finds a package by name in the index\nthe builds use and lists its versions. For Python it marks the versions that have no\nwheel for the environment's Python: pip would compile those during the build, which\noften fails. ",[24,1054,1055],{},"Add to the list"," writes the line.",[14,1058,1059],{},"Pin the versions: a line without one takes whatever is newest when the image is\nbuilt. Tabularia refuses a line that is not a package name and version, and says which\nline it is.",[14,1061,1062,1063,1066],{},"Saving builds ",[24,1064,1065],{},"version 1",". It goes through these states:",[1068,1069,1070,1075,1081,1087],"ol",{},[21,1071,1072,345],{},[24,1073,1074],{},"Waiting to build",[21,1076,1077,1080],{},[24,1078,1079],{},"Building",": a Job in the build namespace builds the image and pushes it.",[21,1082,1083,1086],{},[24,1084,1085],{},"Checking",": a Job in the scripts namespace runs the new image, by digest, as a\nscript would, loads the packages a script needs to read its input, and lists every\ninstalled package.",[21,1088,1089,1092],{},[24,1090,1091],{},"Ready",": the version is in use.",[14,1094,1095],{},"The list of installed packages, the end of the build log and the image reference stay\nwith the version. A build that fails says why and keeps the version in use as it was.",[14,1097,1098],{},"Changing the version or the packages builds a new version. Changing anything else\napplies to the next run without a build:",[18,1100,1101,1104,1107,1110],{},[21,1102,1103],{},"the description;",[21,1105,1106],{},"the resources (CPU and memory requested and limited, the longest run);",[21,1108,1109],{},"who can use the environment;",[21,1111,1112],{},"the variables.",[14,1114,1115,1118,1119,1122],{},[24,1116,1117],{},"Build again"," makes a new version from the same packages: after a failure, or to\nstart from a newer base image. ",[24,1120,1121],{},"Use this version"," puts an older version back; its\nimage is still in the registry.",[483,1124,1126],{"id":1125},"who-can-use-an-environment","Who can use an environment",[14,1128,1129],{},"People in the environment's groups who can edit a flow can add Script nodes with it.\nAdministrators can use every environment.",[14,1131,1132,1133,1136],{},"Someone outside those groups ",[24,1134,1135],{},"cannot open a flow with a Script node of the\nenvironment",": its code is not theirs to read. The flow is still listed in Explore, in\nFlows and in the search, greyed out with a lock and the reason on hover; it is not a\nlink and has no actions. Reached by its address, the editor says the same in place of\nthe canvas. Its versions, a copy, a run, a schedule and adding it to a group of flows\nare refused the same way. An environment that no longer exists blocks nobody.",[483,1138,1140],{"id":1139},"variables","Variables",[14,1142,1143,1144,1147],{},"The variables are what the scripts find in their environment, and they are separate\nfrom Tabularia's. A ",[24,1145,1146],{},"secret"," variable is encrypted, never shown again, and given to\nthe scripts like the others.",[14,1149,1150,1151,1154,1155,1158],{},"Names starting with ",[198,1152,1153],{},"TABULARIA_"," or ",[198,1156,1157],{},"AWS_"," are reserved for what Tabularia sets in\nevery Job: where the input and the output are, and the bucket's key.",[14,1160,1161,1164],{},[24,1162,1163],{},"From Tabularia"," is the exception. It hands the scripts the value of one of\nTabularia's own variables, named in the field and read when each script starts.\nEveryone who writes scripts with that environment can read it. The panel says so, and\nthe audit records who set it.",[95,1166,32],{"id":1167},"the-script-node",[14,1169,1170,1171,1173,1174,345],{},"In the flow editor, ",[24,1172,89],{}," is in the Control group of the palette for whoever may\nuse at least one environment. Choose its environment in the node panel, then ",[24,1175,1176],{},"Edit\nthe code",[483,1178,1180],{"id":1179},"input-and-output-come-from-the-edges","Input and output come from the edges",[100,1182,1183,1193],{},[103,1184,1185],{},[106,1186,1187,1190],{},[109,1188,1189],{},"the node has",[109,1191,1192],{},"the script gets",[118,1194,1195,1206,1217],{},[106,1196,1197,1200],{},[123,1198,1199],{},"a data edge coming in",[123,1201,1202,1205],{},[198,1203,1204],{},"TABULARIA_INPUT",": the step before it, one parquet file in the scripts bucket",[106,1207,1208,1211],{},[123,1209,1210],{},"a data edge going out",[123,1212,1213,1216],{},[198,1214,1215],{},"TABULARIA_OUTPUT",": where it must write one parquet file; the nodes after it read that",[106,1218,1219,1222],{},[123,1220,1221],{},"neither",[123,1223,1224],{},"nothing to read or write: it runs as a step of the flow, for instance to start something outside Tabularia",[14,1226,1227,1228,1231],{},"Polars, pyarrow and R's arrow read and write those ",[198,1229,1230],{},"s3:\u002F\u002F"," addresses by themselves:",[191,1233,1237],{"className":1234,"code":1235,"language":1236,"meta":196,"style":196},"language-python shiki shiki-themes github-dark-dimmed github-light","import os\nimport polars as pl\n\ndf = pl.scan_parquet(os.environ[\"TABULARIA_INPUT\"])\ndf.with_columns(score=pl.col(\"amount\") * 2).sink_parquet(os.environ[\"TABULARIA_OUTPUT\"])\n","python",[198,1238,1239,1247,1260,1266,1283],{"__ignoreMap":196},[201,1240,1241,1244],{"class":203,"line":204},[201,1242,1243],{"class":367},"import",[201,1245,1246],{"class":211}," os\n",[201,1248,1249,1251,1254,1257],{"class":203,"line":215},[201,1250,1243],{"class":367},[201,1252,1253],{"class":211}," polars ",[201,1255,1256],{"class":367},"as",[201,1258,1259],{"class":211}," pl\n",[201,1261,1262],{"class":203,"line":228},[201,1263,1265],{"emptyLinePlaceholder":1264},true,"\n",[201,1267,1268,1271,1274,1277,1280],{"class":203,"line":236},[201,1269,1270],{"class":211},"df ",[201,1272,1273],{"class":367},"=",[201,1275,1276],{"class":211}," pl.scan_parquet(os.environ[",[201,1278,1279],{"class":244},"\"TABULARIA_INPUT\"",[201,1281,1282],{"class":211},"])\n",[201,1284,1285,1288,1292,1294,1297,1300,1303,1306,1309,1312,1315],{"class":203,"line":252},[201,1286,1287],{"class":211},"df.with_columns(",[201,1289,1291],{"class":1290},"sNjOc","score",[201,1293,1273],{"class":367},[201,1295,1296],{"class":211},"pl.col(",[201,1298,1299],{"class":244},"\"amount\"",[201,1301,1302],{"class":211},") ",[201,1304,1305],{"class":367},"*",[201,1307,1308],{"class":224}," 2",[201,1310,1311],{"class":211},").sink_parquet(os.environ[",[201,1313,1314],{"class":244},"\"TABULARIA_OUTPUT\"",[201,1316,1282],{"class":211},[191,1318,1322],{"className":1319,"code":1320,"language":1321,"meta":196,"style":196},"language-r shiki shiki-themes github-dark-dimmed github-light","library(arrow)\ndf \u003C- read_parquet(Sys.getenv(\"TABULARIA_INPUT\"))\nwrite_parquet(df, Sys.getenv(\"TABULARIA_OUTPUT\"))\n","r",[198,1323,1324,1329,1334],{"__ignoreMap":196},[201,1325,1326],{"class":203,"line":204},[201,1327,1328],{},"library(arrow)\n",[201,1330,1331],{"class":203,"line":215},[201,1332,1333],{},"df \u003C- read_parquet(Sys.getenv(\"TABULARIA_INPUT\"))\n",[201,1335,1336],{"class":203,"line":228},[201,1337,1338],{},"write_parquet(df, Sys.getenv(\"TABULARIA_OUTPUT\"))\n",[14,1340,1341],{},"A script that has nodes after it and writes nothing fails.",[483,1343,1345],{"id":1344},"run-and-the-preview-of-the-nodes-after-it","Run, and the preview of the nodes after it",[14,1347,1348,1350,1351,1354],{},[24,1349,155],{}," in the node panel runs the node on its own, as the flow is saved (with unsaved\nchanges the button is ",[24,1352,1353],{},"Save and run","):",[18,1356,1357,1360,1363],{},[21,1358,1359],{},"the chain before it runs on the sources' development sample;",[21,1361,1362],{},"the nodes after it do not run, and nothing is written to an Output;",[21,1364,1365],{},"it runs in the trials' namespace, within its quota, never among the flows' runs.",[14,1367,1368],{},"It takes a few seconds once the environment's image is on the cluster's nodes. Its\nlog and its outcome show in the panel, and the run page lists it as a trial.",[14,1370,1371],{},"The nodes after a Script node preview what it wrote last: its latest trial or the\nlatest run of the flow, whichever is newer. The preview says which, and when; if the\nsaved code has changed since, it says so too. Before any of them, those nodes have no\npreview.",[483,1373,1375],{"id":1374},"the-code","The code",[14,1377,1378],{},"The editor is Monaco, the editor of VS Code. On the left, the node's files: the one\nmarked ▶ is the one that runs, the others sit next to it in the same folder.",[18,1380,1381,1391],{},[21,1382,1383,1384,1387,1388,345],{},"In Python, ",[198,1385,1386],{},"lib\u002Fscoring.py"," is imported as ",[198,1389,1390],{},"from lib import scoring",[21,1392,1393,1394,1397,1398,345],{},"In R, ",[198,1395,1396],{},"source(\"R\u002Fhelpers.R\")"," reads ",[198,1399,1400],{},"R\u002Fhelpers.R",[14,1402,1403],{},"On the right, what the script finds in its environment: Tabularia's variables, the\nenvironment's variables (secret ones by name only), the installed packages. A click\ninserts one at the cursor.",[14,1405,1406],{},[70,1407],{"alt":1408,"src":1409},"The code editor of the Script node «Margin trend», Python 3.12: on the left the node's files, main.py marked as the one that runs, and a field to add another such as lib\u002Fhelpers.py; in the middle the code of the example below; on the right the variables set by Tabularia, TABULARIA_INPUT and TABULARIA_OUTPUT first, and the installed packages; under the code, «Last run, now: succeeded»","\u002Fmedia\u002Fguide\u002Fscripting\u002Fscript-code-editor.png",[14,1411,1412],{},[77,1413,1414,1415,1419],{},"The code editor of the ",[41,1416,1418],{"href":1417},"#example-the-margin-trend-of-each-category","example below",":\nfiles on the left, the variables and packages the script finds on the right, the outcome\nof the last run under the code.",[14,1421,1422],{},"While typing, the editor offers:",[18,1424,1425,1437,1445],{},[21,1426,1427,1428,301,1431,341,1434,660],{},"the variables, inside ",[198,1429,1430],{},"os.environ[\"…\"]",[198,1432,1433],{},"os.getenv(\"…\")",[198,1435,1436],{},"Sys.getenv(\"…\")",[21,1438,1439,1440,341,1442,660],{},"the installed modules after ",[198,1441,1243],{},[198,1443,1444],{},"library(",[21,1446,1447,1448,1451,1452,1455,1456,345],{},"the public names of a module after its alias (",[198,1449,1450],{},"pl."," after ",[198,1453,1454],{},"import polars as pl",")\nor after ",[198,1457,1458],{},"pkg::",[14,1460,1461,1464,1465,1471,1472,1474,1475,1478],{},[24,1462,1463],{},"Problems, before running."," Python code is checked by ",[41,1466,1470],{"href":1467,"rel":1468},"https:\u002F\u002Fgithub.com\u002Fastral-sh\u002Fruff",[1469],"nofollow","Ruff","\nin the gateway when typing pauses. It underlines names used and never defined or\nimported, syntax errors, and imports or variables left unused; the list of files\ncounts each file's problems. An ",[198,1473,1243],{}," of a module that is neither installed in\nthe environment (the list its build's check wrote), nor part of Python, nor one of\nthe node's files is underlined too. ",[198,1476,1477],{},"SCRIPTS__LINT=false"," turns the check off. R code\nis not checked.",[14,1480,1481,1484,1485,1488,1489,1492],{},[24,1482,1483],{},"The last run."," Under the code is the log of the node's last run. When it failed,\nthe editor marks the line it stopped at, in the file where it stopped, until that\nfile changes, and ",[24,1486,1487],{},"Go to"," opens it. For R, only a file read with ",[198,1490,1491],{},"source()"," that\ndoes not parse gives a line.",[100,1494,1495,1504],{},[103,1496,1497],{},[106,1498,1499,1502],{},[109,1500,1501],{},"variable",[109,1503],{},[118,1505,1506,1517,1527,1537,1553,1563],{},[106,1507,1508,1514],{},[123,1509,1510,301,1512],{},[198,1511,1204],{},[198,1513,1215],{},[123,1515,1516],{},"the step's files, or empty",[106,1518,1519,1524],{},[123,1520,1521],{},[198,1522,1523],{},"TABULARIA_STEP_ID",[123,1525,1526],{},"the step's random id",[106,1528,1529,1534],{},[123,1530,1531],{},[198,1532,1533],{},"TABULARIA_RUN_ID",[123,1535,1536],{},"the flow's run",[106,1538,1539,1550],{},[123,1540,1541,301,1544,301,1547],{},[198,1542,1543],{},"TABULARIA_FLOW_ID",[198,1545,1546],{},"TABULARIA_FLOW_NAME",[198,1548,1549],{},"TABULARIA_NODE_ID",[123,1551,1552],{},"where the script is",[106,1554,1555,1560],{},[123,1556,1557],{},[198,1558,1559],{},"TABULARIA_ATTEMPT",[123,1561,1562],{},"1, then 2… when the node retries",[106,1564,1565,1570],{},[123,1566,1567],{},[198,1568,1569],{},"AWS_*",[123,1571,1572],{},"the scripts bucket's key and endpoint, for the libraries",[14,1574,1575,1578,1579,341,1581,345],{},[24,1576,1577],{},"The path check."," The people who write scripts are trusted, and the scripts may\ncall any service. One thing is checked when the flow is saved: the code may not name\nthe scripts bucket, where the files of the other runs are. Read and write through\n",[198,1580,1204],{},[198,1582,1215],{},[483,1584,1586],{"id":1585},"the-assistant","The assistant",[14,1588,1589,1590,301,1594,1597,1598,1601,1602,1605,1606,1609,1610,1613],{},"Where the installation has an ",[41,1591,1593],{"href":1592},"\u002Fdocs\u002Fintegrations\u002Fai","AI assistant",[24,1595,1596],{},"Assistant","\nin the editor's header puts it beside the code, in place of the environment's panel.\nAsk for a change, or a script that does something with the inputs: it answers and may\n",[24,1599,1600],{},"propose files",", whole, listed with what each change adds and removes. ",[24,1603,1604],{},"Apply"," writes\nthem in the editor as one change (Ctrl+Z in a file takes it back); ",[24,1607,1608],{},"Discard"," leaves the\ncode as it is. It never saves nor runs the script. After a failed run, ",[24,1611,1612],{},"Explain and\nfix the last error"," asks it why, with the error.",[14,1615,1616,1619,1620,1624,1625,1628],{},[24,1617,1618],{},"What reaches the model:"," the code, the environment's language, version and packages,\nthe names of the variables the script finds, the names and types of its inputs'\ncolumns, and, for a failed run, only the error of its log (the Python traceback, the R\nerror): never a row of data, nor anything else the script printed. Its conversations\nare saved as the other assistants', with the same daily caps. With\n",[41,1621,1623],{"href":1622},"\u002Fdocs\u002Fintegrations\u002Fai#metadata-only-mode","metadata-only mode"," on, the error stays out too,\nsince it may quote a value: the model gets the exit code, the editor says so, and ",[24,1626,1627],{},"Explain\nand fix the last error"," is not offered.",[483,1630,1632],{"id":1631},"example-the-margin-trend-of-each-category","Example: the margin trend of each category",[14,1634,1635,1636,1639,1640,301,1643,301,1646,301,1649,730,1652,1655],{},"The sample datasource «Margin by Category and Month» (",[24,1637,1638],{},"Sample \u002F Private tables",") has one\nrow per product category and month: ",[198,1641,1642],{},"categoria",[198,1644,1645],{},"anno_mese",[198,1647,1648],{},"ricavo",[198,1650,1651],{},"margine",[198,1653,1654],{},"margine_pct"," and more. This Script node adds, for each row, the margin of the last three\nmonths and the revenue against the same month a year before, which a flow's operations\nwould need several steps to compute.",[1068,1657,1658,1669,1680,1694,1700,1720,1729],{},[21,1659,1660,1661,1664,1665,1668],{},"In Explore, open a folder you may edit (here «Sales analysis»), ",[24,1662,1663],{},"New"," → ",[24,1666,1667],{},"New flow",",\nand name it «Margin trend».",[21,1670,1671,1672,1675,1676,1679],{},"Select the ",[24,1673,1674],{},"Source"," node and, in its panel, ",[24,1677,1678],{},"or use a catalog datasource"," →\n«Margin by Category and Month».",[21,1681,1682,1683,1685,1686,1689,1690,1693],{},"Drag ",[24,1684,89],{}," from the palette's ",[24,1687,1688],{},"Control"," group next to it, and connect the\nSource to it. In its panel: name «Margin trend», ",[24,1691,1692],{},"Written here",", environment\n«Python».",[21,1695,1682,1696,1699],{},[24,1697,1698],{},"Output (datasource \u002F database)"," after it, connect the Script node to it, and\nname the datasource «Margin trend by category» in your folder.",[21,1701,1702,1705,1706,1709,1710,1713,1714,697,1717,1719],{},[24,1703,1704],{},"Save",". In the Script node's panel, ",[24,1707,1708],{},"Edit the code",", replace ",[198,1711,1712],{},"main.py"," with the code\nbelow, ",[24,1715,1716],{},"Close",[24,1718,1704],{}," again.",[21,1721,1722,1724,1725,1728],{},[24,1723,155],{}," in the Script node's panel. In a few seconds the panel says «Last run from here\nnow: SUCCESS · 184 rows»; ",[24,1726,1727],{},"Log"," shows what the script printed, and the preview under\nthe canvas shows the rows it wrote, newest month first.",[21,1730,1731,1733],{},[24,1732,155],{}," in the toolbar runs the whole flow and publishes «Margin trend by category».",[191,1735,1737],{"className":1234,"code":1736,"language":1236,"meta":196,"style":196},"import os\n\nimport polars as pl\n\n# the step before this node: Margin by Category and Month\nmargin = pl.read_parquet(os.environ[\"TABULARIA_INPUT\"])\nprint(f\"read {margin.height} rows\")\n\nrevenue_3m = pl.col(\"ricavo\").rolling_sum(3).over(\"categoria\")\nmargin_3m = pl.col(\"margine\").rolling_sum(3).over(\"categoria\")\nyear_ago = pl.col(\"ricavo\").shift(12).over(\"categoria\")\ngrowth = pl.col(\"ricavo\") \u002F year_ago - 1\n\ntrend = (\n    margin.sort(\"categoria\", \"anno_mese\")\n    .with_columns(\n        margin_pct_3m=(margin_3m \u002F revenue_3m * 100).round(2),\n        revenue_vs_last_year_pct=(growth * 100).round(1),\n    )\n    .select(\"categoria\", \"anno_mese\", pl.col(\"ricavo\").round(0),\n            \"margine_pct\", \"margin_pct_3m\",\n            \"revenue_vs_last_year_pct\")\n    .sort(\"anno_mese\", \"categoria\", descending=[True, False])\n)\ntrend.write_parquet(os.environ[\"TABULARIA_OUTPUT\"])\nprint(f\"wrote {trend.height} rows\")\n",[198,1738,1739,1745,1749,1759,1763,1768,1782,1811,1815,1843,1866,1890,1916,1921,1932,1947,1953,1983,2005,2011,2035,2048,2056,2088,2093,2103],{"__ignoreMap":196},[201,1740,1741,1743],{"class":203,"line":204},[201,1742,1243],{"class":367},[201,1744,1246],{"class":211},[201,1746,1747],{"class":203,"line":215},[201,1748,1265],{"emptyLinePlaceholder":1264},[201,1750,1751,1753,1755,1757],{"class":203,"line":228},[201,1752,1243],{"class":367},[201,1754,1253],{"class":211},[201,1756,1256],{"class":367},[201,1758,1259],{"class":211},[201,1760,1761],{"class":203,"line":236},[201,1762,1265],{"emptyLinePlaceholder":1264},[201,1764,1765],{"class":203,"line":252},[201,1766,1767],{"class":248},"# the step before this node: Margin by Category and Month\n",[201,1769,1770,1773,1775,1778,1780],{"class":203,"line":266},[201,1771,1772],{"class":211},"margin ",[201,1774,1273],{"class":367},[201,1776,1777],{"class":211}," pl.read_parquet(os.environ[",[201,1779,1279],{"class":244},[201,1781,1282],{"class":211},[201,1783,1784,1787,1790,1793,1796,1799,1802,1805,1808],{"class":203,"line":787},[201,1785,1786],{"class":224},"print",[201,1788,1789],{"class":211},"(",[201,1791,1792],{"class":367},"f",[201,1794,1795],{"class":244},"\"read ",[201,1797,1798],{"class":395},"{",[201,1800,1801],{"class":211},"margin.height",[201,1803,1804],{"class":395},"}",[201,1806,1807],{"class":244}," rows\"",[201,1809,1810],{"class":211},")\n",[201,1812,1813],{"class":203,"line":793},[201,1814,1265],{"emptyLinePlaceholder":1264},[201,1816,1818,1821,1823,1826,1829,1832,1835,1838,1841],{"class":203,"line":1817},9,[201,1819,1820],{"class":211},"revenue_3m ",[201,1822,1273],{"class":367},[201,1824,1825],{"class":211}," pl.col(",[201,1827,1828],{"class":244},"\"ricavo\"",[201,1830,1831],{"class":211},").rolling_sum(",[201,1833,1834],{"class":224},"3",[201,1836,1837],{"class":211},").over(",[201,1839,1840],{"class":244},"\"categoria\"",[201,1842,1810],{"class":211},[201,1844,1846,1849,1851,1853,1856,1858,1860,1862,1864],{"class":203,"line":1845},10,[201,1847,1848],{"class":211},"margin_3m ",[201,1850,1273],{"class":367},[201,1852,1825],{"class":211},[201,1854,1855],{"class":244},"\"margine\"",[201,1857,1831],{"class":211},[201,1859,1834],{"class":224},[201,1861,1837],{"class":211},[201,1863,1840],{"class":244},[201,1865,1810],{"class":211},[201,1867,1869,1872,1874,1876,1878,1881,1884,1886,1888],{"class":203,"line":1868},11,[201,1870,1871],{"class":211},"year_ago ",[201,1873,1273],{"class":367},[201,1875,1825],{"class":211},[201,1877,1828],{"class":244},[201,1879,1880],{"class":211},").shift(",[201,1882,1883],{"class":224},"12",[201,1885,1837],{"class":211},[201,1887,1840],{"class":244},[201,1889,1810],{"class":211},[201,1891,1893,1896,1898,1900,1902,1904,1907,1910,1913],{"class":203,"line":1892},12,[201,1894,1895],{"class":211},"growth ",[201,1897,1273],{"class":367},[201,1899,1825],{"class":211},[201,1901,1828],{"class":244},[201,1903,1302],{"class":211},[201,1905,1906],{"class":367},"\u002F",[201,1908,1909],{"class":211}," year_ago ",[201,1911,1912],{"class":367},"-",[201,1914,1915],{"class":224}," 1\n",[201,1917,1919],{"class":203,"line":1918},13,[201,1920,1265],{"emptyLinePlaceholder":1264},[201,1922,1924,1927,1929],{"class":203,"line":1923},14,[201,1925,1926],{"class":211},"trend ",[201,1928,1273],{"class":367},[201,1930,1931],{"class":211}," (\n",[201,1933,1935,1938,1940,1942,1945],{"class":203,"line":1934},15,[201,1936,1937],{"class":211},"    margin.sort(",[201,1939,1840],{"class":244},[201,1941,301],{"class":211},[201,1943,1944],{"class":244},"\"anno_mese\"",[201,1946,1810],{"class":211},[201,1948,1950],{"class":203,"line":1949},16,[201,1951,1952],{"class":211},"    .with_columns(\n",[201,1954,1956,1959,1961,1964,1966,1969,1971,1974,1977,1980],{"class":203,"line":1955},17,[201,1957,1958],{"class":1290},"        margin_pct_3m",[201,1960,1273],{"class":367},[201,1962,1963],{"class":211},"(margin_3m ",[201,1965,1906],{"class":367},[201,1967,1968],{"class":211}," revenue_3m ",[201,1970,1305],{"class":367},[201,1972,1973],{"class":224}," 100",[201,1975,1976],{"class":211},").round(",[201,1978,1979],{"class":224},"2",[201,1981,1982],{"class":211},"),\n",[201,1984,1986,1989,1991,1994,1996,1998,2000,2003],{"class":203,"line":1985},18,[201,1987,1988],{"class":1290},"        revenue_vs_last_year_pct",[201,1990,1273],{"class":367},[201,1992,1993],{"class":211},"(growth ",[201,1995,1305],{"class":367},[201,1997,1973],{"class":224},[201,1999,1976],{"class":211},[201,2001,2002],{"class":224},"1",[201,2004,1982],{"class":211},[201,2006,2008],{"class":203,"line":2007},19,[201,2009,2010],{"class":211},"    )\n",[201,2012,2014,2017,2019,2021,2023,2026,2028,2030,2033],{"class":203,"line":2013},20,[201,2015,2016],{"class":211},"    .select(",[201,2018,1840],{"class":244},[201,2020,301],{"class":211},[201,2022,1944],{"class":244},[201,2024,2025],{"class":211},", pl.col(",[201,2027,1828],{"class":244},[201,2029,1976],{"class":211},[201,2031,2032],{"class":224},"0",[201,2034,1982],{"class":211},[201,2036,2038,2041,2043,2046],{"class":203,"line":2037},21,[201,2039,2040],{"class":244},"            \"margine_pct\"",[201,2042,301],{"class":211},[201,2044,2045],{"class":244},"\"margin_pct_3m\"",[201,2047,730],{"class":211},[201,2049,2051,2054],{"class":203,"line":2050},22,[201,2052,2053],{"class":244},"            \"revenue_vs_last_year_pct\"",[201,2055,1810],{"class":211},[201,2057,2059,2062,2064,2066,2068,2070,2073,2075,2078,2081,2083,2086],{"class":203,"line":2058},23,[201,2060,2061],{"class":211},"    .sort(",[201,2063,1944],{"class":244},[201,2065,301],{"class":211},[201,2067,1840],{"class":244},[201,2069,301],{"class":211},[201,2071,2072],{"class":1290},"descending",[201,2074,1273],{"class":367},[201,2076,2077],{"class":211},"[",[201,2079,2080],{"class":224},"True",[201,2082,301],{"class":211},[201,2084,2085],{"class":224},"False",[201,2087,1282],{"class":211},[201,2089,2091],{"class":203,"line":2090},24,[201,2092,1810],{"class":211},[201,2094,2096,2099,2101],{"class":203,"line":2095},25,[201,2097,2098],{"class":211},"trend.write_parquet(os.environ[",[201,2100,1314],{"class":244},[201,2102,1282],{"class":211},[201,2104,2106,2108,2110,2112,2115,2117,2120,2122,2124],{"class":203,"line":2105},26,[201,2107,1786],{"class":224},[201,2109,1789],{"class":211},[201,2111,1792],{"class":367},[201,2113,2114],{"class":244},"\"wrote ",[201,2116,1798],{"class":395},[201,2118,2119],{"class":211},"trend.height",[201,2121,1804],{"class":395},[201,2123,1807],{"class":244},[201,2125,1810],{"class":211},[14,2127,2128],{},[70,2129],{"alt":2130,"src":2131},"The flow «Margin trend» in the editor: Source «Margin by Category and Month», the Script node «Margin trend» (Python) and a datasource output. The Script node is selected; Run in its panel shows «Running…», then «Last run from here now: SUCCESS · 184 rows»; the Log opens on «read 184 rows» and «wrote 184 rows», and the preview below lists categoria, anno_mese, ricavo, margine_pct, margin_pct_3m and revenue_vs_last_year_pct, newest month first","\u002Fmedia\u002Fguide\u002Fscripting\u002Fscript-node-run.gif",[14,2133,2134],{},[77,2135,2136],{},"Run in the Script node's panel: the trial runs on the cluster, its log arrives in the\npanel, and what it wrote becomes the preview of the nodes after it.",[14,2138,2139,2140,2143,2144,345],{},"The newest month can show a large drop against the year before: in the sample data it\nholds only part of the month. The months of the first year have no\n",[198,2141,2142],{},"revenue_vs_last_year_pct",", and the first two of each category no ",[198,2145,2146],{},"margin_pct_3m",[483,2148,2150],{"id":2149},"running","Running",[14,2152,2153],{},"A flow with a Script node runs as an orchestration, like one with a Container node.",[18,2155,2156,2162,2176],{},[21,2157,2158,2161],{},[24,2159,2160],{},"Order."," The node runs after the steps it reads from and before the ones that\nread from it, with or without a sequence edge.",[21,2163,2164,2167,2168,2171,2172,2175],{},[24,2165,2166],{},"Limits."," ",[24,2169,2170],{},"Timeout"," stops it; ",[24,2173,2174],{},"Retries"," runs it again, as a new Job, after the\ndelay.",[21,2177,2178,2181],{},[24,2179,2180],{},"Failure."," A failure stops the flow: the steps after it would read nothing. A\nscript that goes over the environment's memory limit is stopped by the cluster; the\nrun says so, with the limit (exit code 137: some clusters, kind among them, report it\nas a plain kill, and the run then says that the limit is the likeliest cause).",[14,2183,2184,2185,2188,2189,301,2192,2195],{},"The run page lists the node as a run of its own. It shows the environment and its\nversion, the attempts, the exit code, the variables it was given (secret ones by\nname only) and the end of its log: the last 256 KB (",[198,2186,2187],{},"SCRIPTS__LOG_MAX_BYTES","), read\nwhen the Job ends and kept in Tabularia's database. After 30 days\n(",[198,2190,2191],{},"ORCHESTRATOR__LOG_RETENTION_DAYS",[198,2193,2194],{},"orchestrator.logRetentionDays"," in the chart; 0\nkeeps them) the log is emptied and the run stays, with its outcome, attempts and exit\ncode.",[14,2197,2198],{},"Whoever runs the flow, or the author of its schedule, must be allowed to use the\nenvironment. A schedule set by someone else is switched off when its code changes,\nas for any change that makes a flow reach further.",[95,2200,38],{"id":2201},"saved-scripts",[14,2203,2204,2205,1018,2208,2211,2212,2214,2215,2218],{},"A script can also live on its own, in a folder next to flows and views: ",[24,2206,2207],{},"Explore →\nNew → Script (Python or R)",[24,2209,2210],{},"New script"," in the ",[24,2213,82],{}," tab. It is written\nin the same editor as the Script node, in one of the environments you may use, and it\nkeeps its ",[24,2216,2217],{},"versions",": every save that changes it is one; an older one can be looked\nat and restored, and becomes the newest.",[14,2220,2221],{},[70,2222],{"alt":2223,"src":2224},"The page of the saved script «Margins under target» in Sales analysis: Schedule, Run and Save at the top; the Python environment and the description; the code, main.py, 13 lines, with Edit the code; the input MARGIN, the datasource «Margin by Category and Month» from Sample › Private tables; the output, published as the datasource «Margins under target» in Sales analysis and replaced at every run; the versions on the right","\u002Fmedia\u002Fguide\u002Fscripting\u002Fsaved-script-page.png",[14,2226,2227],{},[77,2228,2229],{},"A saved script's page: its environment, its code, the datasources it reads under their\naliases, and where its result goes.",[14,2231,2232,2235,2236,2239],{},[24,2233,2234],{},"Who sees it."," VIEW on its folder ",[24,2237,2238],{},"and"," the use of its environment. Whoever lacks\neither does not see it at all: not in Explore, not in the search, not in the Scripting\ntab, not in the pickers of flows and groups. RUN on the folder runs and schedules it,\nEDIT changes, moves and deletes it. Administrators see every script.",[483,2241,2243],{"id":2242},"inputs","Inputs",[14,2245,2246,2247,2250,2251,2254,2255,2258,2259,2262],{},"A saved script reads the datasources you declare on its page, each with an ",[24,2248,2249],{},"alias",".\nThe alias is proposed from the datasource's name (",[198,2252,2253],{},"Vendite Nord 2024"," →\n",[198,2256,2257],{},"VENDITE_NORD_2024","; for two datasources with the same name in two folders, the\nfolder's name is added) and can be changed: capitals, digits and ",[198,2260,2261],{},"_",", unique in the\nscript. The script keeps the datasource itself, not its name or its file: renaming,\nmoving or refreshing the datasource changes nothing for the script, and every run reads\nits data as it is at that moment.",[100,2264,2265,2274],{},[103,2266,2267],{},[106,2268,2269,2272],{},[109,2270,2271],{},"Variable",[109,2273],{},[118,2275,2276,2288,2298,2307],{},[106,2277,2278,2283],{},[123,2279,2280],{},[198,2281,2282],{},"TABULARIA_INPUT_\u003CALIAS>",[123,2284,128,2285,2287],{},[198,2286,1230],{}," address of that datasource's data, as parquet",[106,2289,2290,2295],{},[123,2291,2292],{},[198,2293,2294],{},"TABULARIA_INPUTS",[123,2296,2297],{},"every input as JSON, keyed by alias: its datasource's id, real name, folder, address and columns",[106,2299,2300,2304],{},[123,2301,2302],{},[198,2303,1215],{},[123,2305,2306],{},"where the script may write its result, as parquet",[106,2308,2309,2326],{},[123,2310,2311,301,2314,301,2317,301,2320,301,2322,301,2324],{},[198,2312,2313],{},"TABULARIA_SCRIPT_ID",[198,2315,2316],{},"TABULARIA_SCRIPT_NAME",[198,2318,2319],{},"TABULARIA_SCRIPT_VERSION",[198,2321,1533],{},[198,2323,1523],{},[198,2325,1559],{},[123,2327,2328],{},"which script, version, run and attempt this is",[191,2330,2332],{"className":1234,"code":2331,"language":1236,"meta":196,"style":196},"import json\nimport os\n\nimport polars as pl\n\ninputs = json.loads(os.environ[\"TABULARIA_INPUTS\"])\nsales = pl.scan_parquet(os.environ[\"TABULARIA_INPUT_VENDITE_NORD_2024\"])\nprint(\"reading\", inputs[\"VENDITE_NORD_2024\"][\"name\"], \"from\", inputs[\"VENDITE_NORD_2024\"][\"folder\"])\nsales.group_by(\"region\").agg(pl.col(\"amount\").sum()).sink_parquet(os.environ[\"TABULARIA_OUTPUT\"])\n",[198,2333,2334,2341,2347,2351,2361,2365,2380,2394,2432],{"__ignoreMap":196},[201,2335,2336,2338],{"class":203,"line":204},[201,2337,1243],{"class":367},[201,2339,2340],{"class":211}," json\n",[201,2342,2343,2345],{"class":203,"line":215},[201,2344,1243],{"class":367},[201,2346,1246],{"class":211},[201,2348,2349],{"class":203,"line":228},[201,2350,1265],{"emptyLinePlaceholder":1264},[201,2352,2353,2355,2357,2359],{"class":203,"line":236},[201,2354,1243],{"class":367},[201,2356,1253],{"class":211},[201,2358,1256],{"class":367},[201,2360,1259],{"class":211},[201,2362,2363],{"class":203,"line":252},[201,2364,1265],{"emptyLinePlaceholder":1264},[201,2366,2367,2370,2372,2375,2378],{"class":203,"line":266},[201,2368,2369],{"class":211},"inputs ",[201,2371,1273],{"class":367},[201,2373,2374],{"class":211}," json.loads(os.environ[",[201,2376,2377],{"class":244},"\"TABULARIA_INPUTS\"",[201,2379,1282],{"class":211},[201,2381,2382,2385,2387,2389,2392],{"class":203,"line":787},[201,2383,2384],{"class":211},"sales ",[201,2386,1273],{"class":367},[201,2388,1276],{"class":211},[201,2390,2391],{"class":244},"\"TABULARIA_INPUT_VENDITE_NORD_2024\"",[201,2393,1282],{"class":211},[201,2395,2396,2398,2400,2403,2406,2409,2412,2415,2418,2421,2423,2425,2427,2430],{"class":203,"line":793},[201,2397,1786],{"class":224},[201,2399,1789],{"class":211},[201,2401,2402],{"class":244},"\"reading\"",[201,2404,2405],{"class":211},", inputs[",[201,2407,2408],{"class":244},"\"VENDITE_NORD_2024\"",[201,2410,2411],{"class":211},"][",[201,2413,2414],{"class":244},"\"name\"",[201,2416,2417],{"class":211},"], ",[201,2419,2420],{"class":244},"\"from\"",[201,2422,2405],{"class":211},[201,2424,2408],{"class":244},[201,2426,2411],{"class":211},[201,2428,2429],{"class":244},"\"folder\"",[201,2431,1282],{"class":211},[201,2433,2434,2437,2440,2443,2445,2448,2450],{"class":203,"line":1817},[201,2435,2436],{"class":211},"sales.group_by(",[201,2438,2439],{"class":244},"\"region\"",[201,2441,2442],{"class":211},").agg(pl.col(",[201,2444,1299],{"class":244},[201,2446,2447],{"class":211},").sum()).sink_parquet(os.environ[",[201,2449,1314],{"class":244},[201,2451,1282],{"class":211},[14,2453,2454],{},"Every run checks again that whoever runs it (or the schedule's author) may still read\neach input: one they no longer may read, deleted, or still without data fails the run\nbefore anything starts, saying which.",[483,2456,2458],{"id":2457},"output","Output",[14,2460,2461,2462,2464,2465,2468],{},"Optional. What the script writes at ",[198,2463,1215],{}," is kept and shown at the end of\nthe run, on its page; with ",[24,2466,2467],{},"Also publish it as a datasource",", it becomes a datasource\nin the folder you choose (EDIT needed there), replaced at every run or not. When it\nreplaces one, the datasource's data contract, if it has one, is checked first: a broken\nblocking rule leaves the datasource as it was and fails the run, and the refusal is in\nthe contract's history. A description written on the datasource stays across the runs\n(the first one says which script writes it). A script that writes nothing (it sends\nemails, it loads another system) only has its outcome and its log.",[483,2470,2472],{"id":2471},"running-and-scheduling","Running and scheduling",[14,2474,2475,2477,2478,2481],{},[24,2476,155],{},", on the script's page, runs the saved version on the whole data, in the scripts\nnamespace, queued and carried out like a flow's run: one run of a script at a time. The\npage follows it and shows each run's log and the first rows of its output. ",[24,2479,2480],{},"Schedule","\nruns it with a cron, with the authority of whoever set it; as for flows, a change by\nsomeone else that makes the script reach more (other code, other inputs, another\npublish folder, another environment, another folder) switches the schedule off, and\nthe audit log says why.",[14,2483,2484],{},[70,2485],{"alt":2486,"src":2487},"The Runs of a saved script: two runs on the left, versions 2 and 1; the newest succeeded, by hand, in 7 seconds, with a link to the datasource it published; its log says «43 category-months under 25%», and its output, 43 rows, lists anno_mese, categoria, ricavo, margine and margine_pct","\u002Fmedia\u002Fguide\u002Fscripting\u002Fsaved-script-run.png",[14,2489,2490],{},[77,2491,2492],{},"The script's page follows each run: its outcome, its log and the first rows of what it\nwrote.",[483,2494,2496],{"id":2495},"example-the-months-under-target","Example: the months under target",[14,2498,2499],{},"A saved script that reads the sample «Margin by Category and Month» and publishes the\ncategory-months whose margin is under 25%:",[1068,2501,2502,2511,2523,2534,2541,2550],{},[21,2503,2504,2505,1664,2507,2510],{},"In Explore, open your folder (here «Sales analysis»), ",[24,2506,1663],{},[24,2508,2509],{},"Script (Python or\nR)",", name it «Margins under target», environment «Python».",[21,2512,2513,2514,301,2516,2519,2520,345],{},"Under ",[24,2515,2243],{},[24,2517,2518],{},"Add a datasource"," → «Margin by Category and Month», and change\nthe alias it proposes to ",[198,2521,2522],{},"MARGIN",[21,2524,2513,2525,2527,2528,2530,2531,345],{},[24,2526,2458],{},", tick ",[24,2529,2467],{},": name «Margins under\ntarget», folder «Sales analysis», ",[24,2532,2533],{},"Replace it at every run",[21,2535,2536,2538,2539,345],{},[24,2537,1708],{},", write the code below, ",[24,2540,1704],{},[21,2542,2543,2545,2546,2549],{},[24,2544,155],{},": in a few seconds the run succeeds, its log says how many category-months are\nunder 25%, and ",[24,2547,2548],{},"Published as a datasource"," opens the result.",[21,2551,2552,2554],{},[24,2553,2480],{}," runs it every morning, for instance, so the datasource is fresh when people\narrive.",[191,2556,2558],{"className":1234,"code":2557,"language":1236,"meta":196,"style":196},"import os\n\nimport polars as pl\n\nmargin = pl.read_parquet(os.environ[\"TABULARIA_INPUT_MARGIN\"])\nunder = (\n    margin.filter(pl.col(\"margine_pct\") \u003C 25)\n    .select(\"anno_mese\", \"categoria\", pl.col(\"ricavo\", \"margine\").round(0), \"margine_pct\")\n    .sort(\"anno_mese\", \"categoria\", descending=[True, False])\n)\nprint(f\"{under.height} category-months under 25%\")\nunder.write_parquet(os.environ[\"TABULARIA_OUTPUT\"])\n",[198,2559,2560,2566,2570,2580,2584,2597,2606,2624,2653,2679,2683,2706],{"__ignoreMap":196},[201,2561,2562,2564],{"class":203,"line":204},[201,2563,1243],{"class":367},[201,2565,1246],{"class":211},[201,2567,2568],{"class":203,"line":215},[201,2569,1265],{"emptyLinePlaceholder":1264},[201,2571,2572,2574,2576,2578],{"class":203,"line":228},[201,2573,1243],{"class":367},[201,2575,1253],{"class":211},[201,2577,1256],{"class":367},[201,2579,1259],{"class":211},[201,2581,2582],{"class":203,"line":236},[201,2583,1265],{"emptyLinePlaceholder":1264},[201,2585,2586,2588,2590,2592,2595],{"class":203,"line":252},[201,2587,1772],{"class":211},[201,2589,1273],{"class":367},[201,2591,1777],{"class":211},[201,2593,2594],{"class":244},"\"TABULARIA_INPUT_MARGIN\"",[201,2596,1282],{"class":211},[201,2598,2599,2602,2604],{"class":203,"line":266},[201,2600,2601],{"class":211},"under ",[201,2603,1273],{"class":367},[201,2605,1931],{"class":211},[201,2607,2608,2611,2614,2616,2619,2622],{"class":203,"line":787},[201,2609,2610],{"class":211},"    margin.filter(pl.col(",[201,2612,2613],{"class":244},"\"margine_pct\"",[201,2615,1302],{"class":211},[201,2617,2618],{"class":367},"\u003C",[201,2620,2621],{"class":224}," 25",[201,2623,1810],{"class":211},[201,2625,2626,2628,2630,2632,2634,2636,2638,2640,2642,2644,2646,2649,2651],{"class":203,"line":793},[201,2627,2016],{"class":211},[201,2629,1944],{"class":244},[201,2631,301],{"class":211},[201,2633,1840],{"class":244},[201,2635,2025],{"class":211},[201,2637,1828],{"class":244},[201,2639,301],{"class":211},[201,2641,1855],{"class":244},[201,2643,1976],{"class":211},[201,2645,2032],{"class":224},[201,2647,2648],{"class":211},"), ",[201,2650,2613],{"class":244},[201,2652,1810],{"class":211},[201,2654,2655,2657,2659,2661,2663,2665,2667,2669,2671,2673,2675,2677],{"class":203,"line":1817},[201,2656,2061],{"class":211},[201,2658,1944],{"class":244},[201,2660,301],{"class":211},[201,2662,1840],{"class":244},[201,2664,301],{"class":211},[201,2666,2072],{"class":1290},[201,2668,1273],{"class":367},[201,2670,2077],{"class":211},[201,2672,2080],{"class":224},[201,2674,301],{"class":211},[201,2676,2085],{"class":224},[201,2678,1282],{"class":211},[201,2680,2681],{"class":203,"line":1845},[201,2682,1810],{"class":211},[201,2684,2685,2687,2689,2691,2694,2696,2699,2701,2704],{"class":203,"line":1868},[201,2686,1786],{"class":224},[201,2688,1789],{"class":211},[201,2690,1792],{"class":367},[201,2692,2693],{"class":244},"\"",[201,2695,1798],{"class":395},[201,2697,2698],{"class":211},"under.height",[201,2700,1804],{"class":395},[201,2702,2703],{"class":244}," category-months under 25%\"",[201,2705,1810],{"class":211},[201,2707,2708,2711,2713],{"class":203,"line":1892},[201,2709,2710],{"class":211},"under.write_parquet(os.environ[",[201,2712,1314],{"class":244},[201,2714,1282],{"class":211},[483,2716,2718],{"id":2717},"in-flow-groups-and-flows","In flow groups and flows",[18,2720,2721,2727],{},[21,2722,2723,2726],{},[24,2724,2725],{},"A cell of a flow group",": the script runs as a step of its own, with its declared\ninputs; its failure stops the group. Whoever cannot see the script reads a hidden\ncell.",[21,2728,2729,2732,2733,2736,2737,2739,2740,301,2742,2744],{},[24,2730,2731],{},"A Script node by reference",": in the node's panel, ",[24,2734,2735],{},"A saved script"," instead of\n",[24,2738,1692],{},". The node runs the script's current version, with its environment,\ntimeout, retries and declared inputs, next to the node's own edges\n(",[198,2741,1204],{},[198,2743,1215],{},"); the publish setting is ignored there. A fix\nto the script reaches every flow that uses it. Whoever cannot see the script cannot\nopen the flow. A change that makes the script reach more switches off the schedules,\nset by others, of the flows and groups that run it.",[14,2746,2747,2748,2751,2752,2756,2757,2760],{},"A script used by a flow or a group cannot be deleted: the page says which ones use it.\nThe ",[24,2749,2750],{},"Lineage"," shows saved scripts too: the datasources they read, the one they\npublish, the flows and flow groups that run them. With ",[41,2753,2755],{"href":2754},"\u002Fdocs\u002Fopenlineage","OpenLineage","\non, every run of a saved script is a run of the job ",[198,2758,2759],{},"\u003Cfolder>\u002F\u003Cname> (script)"," in\nyour catalog, and a flow's Script node a child job of the flow's run.",[95,2762,2764],{"id":2763},"audit","Audit",[14,2766,2767],{},"Every change is in the audit log: the bucket, the environments, a version put in use,\nevery variable set or removed, and every saved script created, changed, restored, run,\nscheduled or deleted. A variable's value is never recorded.",[2769,2770,2771],"style",{},"html pre.shiki code .shb1k, html code.shiki .shb1k{--shiki-default:#8DDB8C;--shiki-light:#22863A}html pre.shiki code .ssh_m, html code.shiki .ssh_m{--shiki-default:#ADBAC7;--shiki-light:#24292E}html pre.shiki code .swcJU, html code.shiki .swcJU{--shiki-default:#6CB6FF;--shiki-light:#005CC5}html pre.shiki code .sXfbr, html code.shiki .sXfbr{--shiki-default:#96D0FF;--shiki-light:#032F62}html pre.shiki code .sgHix, html code.shiki .sgHix{--shiki-default:#768390;--shiki-light:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html pre.shiki code .sqRhv, html code.shiki .sqRhv{--shiki-default:#F69D50;--shiki-light:#6F42C1}html pre.shiki code .s6PUj, html code.shiki .s6PUj{--shiki-default:#F47067;--shiki-light:#D73A49}html pre.shiki code .sxsTv, html code.shiki .sxsTv{--shiki-default:#F47067;--shiki-light:#005CC5}html pre.shiki code .sJK54, html code.shiki .sJK54{--shiki-default:#8DDB8C;--shiki-light:#005CC5}html pre.shiki code .sNjOc, html code.shiki .sNjOc{--shiki-default:#F69D50;--shiki-light:#E36209}",{"title":196,"searchDepth":215,"depth":228,"links":2773},[2774,2775,2782,2783,2787,2795,2802],{"id":97,"depth":215,"text":98},{"id":185,"depth":215,"text":186,"children":2776},[2777,2778,2779,2780,2781],{"id":485,"depth":228,"text":486},{"id":561,"depth":228,"text":562},{"id":628,"depth":228,"text":629},{"id":635,"depth":228,"text":636},{"id":648,"depth":228,"text":649},{"id":686,"depth":215,"text":687},{"id":972,"depth":215,"text":973,"children":2784},[2785,2786],{"id":1125,"depth":228,"text":1126},{"id":1139,"depth":228,"text":1140},{"id":1167,"depth":215,"text":32,"children":2788},[2789,2790,2791,2792,2793,2794],{"id":1179,"depth":228,"text":1180},{"id":1344,"depth":228,"text":1345},{"id":1374,"depth":228,"text":1375},{"id":1585,"depth":228,"text":1586},{"id":1631,"depth":228,"text":1632},{"id":2149,"depth":228,"text":2150},{"id":2201,"depth":215,"text":38,"children":2796},[2797,2798,2799,2800,2801],{"id":2242,"depth":228,"text":2243},{"id":2457,"depth":228,"text":2458},{"id":2471,"depth":228,"text":2472},{"id":2495,"depth":228,"text":2496},{"id":2717,"depth":228,"text":2718},{"id":2763,"depth":215,"text":2764},"md",{"source":2805},"docs\u002Fscripting.md","\u002Fdocs\u002Fscripting",{"title":5,"description":16},"docs\u002Fscripting","6yYdD004YmkfRxMdfpSvX1qwsXwAACsuSpltLZPLoFI",[2811,2830,2882,2914,2946,2972,2998,3023],{"title":2812,"icon":2813,"intro":2814,"items":2815},"Getting started","start","Install it on one machine, learn the few ideas everything else is built on, and run a first flow.",[2816,2818,2821,2824,2827],{"path":2817,"title":2812},"\u002Fdocs\u002Fgetting-started",{"path":2819,"title":2820},"\u002Fdocs\u002Fgetting-started-docker-compose","Walkthrough with Docker Compose",{"path":2822,"title":2823},"\u002Fdocs\u002Fdeploy\u002Fminikube","Try it on your computer (minikube)",{"path":2825,"title":2826},"\u002Fdocs\u002Fconcepts","Concepts",{"path":2828,"title":2829},"\u002Fdocs\u002Foverview","Overview (README)",{"title":2831,"icon":2832,"intro":2833,"items":2834},"Capabilities","capabilities","What you can build: visual flows, analysis on any step, AI questions, pixel-perfect reports, dashboards, Python, R and SQL scripts, containers and the orchestration that ties them together.",[2835,2838,2841,2844,2847,2850,2853,2856,2858,2861,2862,2865,2867,2870,2873,2876,2879],{"path":2836,"title":2837},"\u002Fdocs\u002Fcapabilities","All capabilities",{"path":2839,"title":2840},"\u002Fdocs\u002Fcapabilities\u002Fflows","Low-code flows",{"path":2842,"title":2843},"\u002Fdocs\u002Fcapabilities\u002Fpreview-and-analysis","Preview and analysis",{"path":2845,"title":2846},"\u002Fdocs\u002Fcapabilities\u002Fask-your-data","Ask your data with AI",{"path":2848,"title":2849},"\u002Fdocs\u002Fcapabilities\u002Freports","Pixel-perfect reports",{"path":2851,"title":2852},"\u002Fdocs\u002Fcapabilities\u002Freport-functions","Report functions",{"path":2854,"title":2855},"\u002Fdocs\u002Fcapabilities\u002Fdashboards","Dashboards",{"path":2806,"title":2857},"Python and R",{"path":2859,"title":2860},"\u002Fdocs\u002Fsql-scripts","SQL scripts",{"path":65,"title":66},{"path":2863,"title":2864},"\u002Fdocs\u002Fcontainer-node","Docker images in flows",{"path":43,"title":2866},"Flow groups: orchestration",{"path":2868,"title":2869},"\u002Fdocs\u002Fcapabilities\u002Fscheduling","Scheduling",{"path":2871,"title":2872},"\u002Fdocs\u002Fpushmail","Pushmail: datasources, reports and dashboards by email",{"path":2874,"title":2875},"\u002Fdocs\u002Fplaceholders","Dynamic dates",{"path":2877,"title":2878},"\u002Fdocs\u002Fcapabilities\u002Fengines","Engines",{"path":2880,"title":2881},"\u002Fdocs\u002Fengines\u002Fengine-differences","Engine differences",{"title":2883,"icon":2884,"intro":2885,"items":2886},"Sources","sources","Every way data comes in: files, databases, buckets, SFTP folders, HTTP APIs, SharePoint, Google Sheets, and the results of your own flows.",[2887,2890,2893,2896,2899,2902,2905,2908,2911],{"path":2888,"title":2889},"\u002Fdocs\u002Fsources","All sources",{"path":2891,"title":2892},"\u002Fdocs\u002Fsources\u002Ffiles","Uploaded files",{"path":2894,"title":2895},"\u002Fdocs\u002Fsources\u002Fdatabases","Database tables and queries",{"path":2897,"title":2898},"\u002Fdocs\u002Fs3-datasource","S3 buckets",{"path":2900,"title":2901},"\u002Fdocs\u002Fsftp","SFTP folders",{"path":2903,"title":2904},"\u002Fdocs\u002Fapi-datasource","HTTP APIs",{"path":2906,"title":2907},"\u002Fdocs\u002Fsources\u002Fsharepoint","Excel on SharePoint",{"path":2909,"title":2910},"\u002Fdocs\u002Fsources\u002Fgoogle-sheets","Google Sheets",{"path":2912,"title":2913},"\u002Fdocs\u002Fsources\u002Fflow-outputs","Results of flows and scripts",{"title":2915,"icon":2916,"intro":2917,"items":2918},"Connections","connections","The systems Tabularia reads from and writes to, how each connection is set up, and who may use it.",[2919,2922,2925,2928,2931,2933,2934,2936,2937,2940,2943],{"path":2920,"title":2921},"\u002Fdocs\u002Fconnections","All connections",{"path":2923,"title":2924},"\u002Fdocs\u002Fconnections\u002Fdatabases","Databases",{"path":2926,"title":2927},"\u002Fdocs\u002Fsqlserver","Microsoft SQL Server driver",{"path":2929,"title":2930},"\u002Fdocs\u002Fconnections\u002Fs3","S3 and compatible storage",{"path":2900,"title":2932},"SFTP servers",{"path":2903,"title":2904},{"path":2906,"title":2935},"SharePoint",{"path":2909,"title":2910},{"path":2938,"title":2939},"\u002Fdocs\u002Fconnections\u002Fsmtp","SMTP email",{"path":2941,"title":2942},"\u002Fdocs\u002Fcontainer-registries","Container registries",{"path":2944,"title":2945},"\u002Fdocs\u002Fconnections\u002Fdestinations","Writing data out",{"title":2947,"icon":2948,"intro":2949,"items":2950},"Integrations","integrations","How Tabularia fits your stack: AI providers, the REST API and webhooks, dbt, OpenLineage, single sign-on, SIEM and monitoring.",[2951,2954,2956,2959,2962,2963,2966,2969],{"path":2952,"title":2953},"\u002Fdocs\u002Fintegrations","All integrations",{"path":1592,"title":2955},"AI: where it is used",{"path":2957,"title":2958},"\u002Fdocs\u002Fapi","API, tokens and webhooks",{"path":2960,"title":2961},"\u002Fdocs\u002Fdbt-export","Export to dbt",{"path":2754,"title":2755},{"path":2964,"title":2965},"\u002Fdocs\u002Fsso-keycloak","Single sign-on with Keycloak",{"path":2967,"title":2968},"\u002Fdocs\u002Fsiem-splunk","Audit log to Splunk",{"path":2970,"title":2971},"\u002Fdocs\u002Fintegrations\u002Fmonitoring","Monitoring",{"title":2973,"icon":2974,"intro":2975,"items":2976},"Security & governance","governance","Who may see and do what, what is recorded, and how data quality and personal data are kept in check.",[2977,2980,2983,2986,2989,2992,2995],{"path":2978,"title":2979},"\u002Fdocs\u002Fgovernance","Overview",{"path":2981,"title":2982},"\u002Fdocs\u002Fgovernance\u002Fpermissions","Permissions",{"path":2984,"title":2985},"\u002Fdocs\u002Fgovernance\u002Faudit-log","Audit log",{"path":2987,"title":2988},"\u002Fdocs\u002Fgovernance\u002Fdata-contracts","Data contracts",{"path":2990,"title":2991},"\u002Fdocs\u002Fdesign\u002Fsso-group-mapping","SSO and group mapping",{"path":2993,"title":2994},"\u002Fdocs\u002Fexamples\u002Fkeycloak","Keycloak example on a laptop",{"path":2996,"title":2997},"\u002Fdocs\u002Fengines\u002Fclickhouse-ai-user","A ClickHouse user for the AI assistant",{"title":2999,"icon":3000,"intro":3001,"items":3002},"Install & operate","operate","Run it in production on Kubernetes, configure it, upgrade it and keep it healthy.",[3003,3006,3008,3011,3014,3017,3020],{"path":3004,"title":3005},"\u002Fdocs\u002Fdeploy\u002Fkubernetes","Kubernetes and Helm",{"path":623,"title":3007},"A company-like cluster on one computer (kind)",{"path":3009,"title":3010},"\u002Fdocs\u002Fdeploy\u002Fupgrading","Upgrading, backups and restore",{"path":3012,"title":3013},"\u002Fdocs\u002Fdeploy\u002Frelease-checklist","Install and upgrade checklist",{"path":3015,"title":3016},"\u002Fdocs\u002Fconfiguration","Configuration reference",{"path":3018,"title":3019},"\u002Fdocs\u002Fengine-settings","Settings from the panel",{"path":3021,"title":3022},"\u002Fdocs\u002Fdatabase-migrations","Database migrations",{"title":3024,"icon":3025,"intro":3026,"items":3027},"Reference","reference","Release process and the remaining reference pages.",[3028],{"path":3029,"title":3030},"\u002Fdocs\u002Frelease-process","Release process",1793656371141]