This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Hi All,I am trying to set up my teams' infrastructure in BQ and
currently, we are looking to have 2 projects 1 for prod and 1 for dev.
and in each of the projects we hope to set up our BQ with a medallion
structure. we hope to have the prod to be a c...
I am currently managing a dataform project with numerous
transformations. These transformations are executed through a
combination of workflow configuration and release configuration for
orchestration, scheduled to run daily, weekly, or monthly, depe...
Hi all,I have the configuration below in my dataform.json file{
"defaultSchema": "datalake_analytics", "assertionSchema":
"dataform_assertions", "warehouse": "bigquery", "defaultDatabase":
"cfc-2017-site-login", "defaultLocation": "EU"} the "defaultL...
I created an incremental Dataform code that when I run as not
incremental, it runs so smooth and perfect but when I run it as
incremental, I get errors. Please see code below and what could be the
cause of the errorconfig { type: "incremental", schem...
Hi guys,I am trying to create an incremental table for a web and app
event tracker but because I have a limit on my Big Query, I run into the
error below"Resources exceeded during query execution: The query could
not be executed in the allotted memor...
Hello @novas and @ms4446 ,What does the Dag_ID stand for? is it the same
as a tag defined in the config file? So currently I have a dataform
project with multiple transformations, I currently run these
transformations using the workflow configuration...
So how do I handle this if I want to work with datasets stored in
multiple locations in GCP? Cos I noticed that dataform only runs for
datasets that are saved in the same location as the default location.
I have tried that before and when I did, it throws an error that says
location is not a parameter.I am currently thinking this feature is not
yet live in dataform.
@ms4446 wrote:config {type: "incremental",bigquery: {partitionBy:
"DATE(timestamp)",clusterBy: ["user_id",
"event_id"]}}SELECT*FROM${ref("your_source_table")}WHEREDATE(timestamp)
> (SELECT MAX(DATE(timestamp)) FROM ${self()}) So I must create the
tab...