How to run airflow script after previous airflow script is complete?
up vote
0
down vote
favorite
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
236
add a comment |
up vote
0
down vote
favorite
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
236
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
236
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
236
asked Nov 8 at 11:29
pepperjohn
236
asked Nov 8 at 11:29
pepperjohn
236
asked Nov 8 at 11:29
pepperjohn
236
asked Nov 8 at 11:29
pepperjohn
236
236
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
up vote
2
down vote
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
answered Nov 9 at 4:27
Meghdeep Ray
2,44821737
2,44821737
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53206852%2fhow-to-run-airflow-script-after-previous-airflow-script-is-complete%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
