How To Get System.getProperties() In Apache Flink Jobs
up vote
0
down vote
favorite
I am running flink standalone cluster that consist of 3 servers. I added "env.java.opts: '-Dmykey=1000'" this settings on my 3 nodes, in this file flink-1.6.0/conf/flink-conf.yaml
When I check java programs, I can see this parameter passed through to my TaskManager and JobManager
/usr/lib/jvm/java-8-oracle/bin/java -Xms15360M -Xmx15360M -XX:MaxDirectMemorySize=8388607T -Dmykey=1000 -Dlog.file=/home/kafka/flink-1.6.0/log/flink-
When I try to print out all properties, iterating System.getProperties(), I cannot fetch my custom properties.
I have libraries that uses property files to get external information, I cannot change my libraries, make them use ParameterTools. I did a lot things, but still could not solve this. Is there any advice for me?
apache-flink flink-streaming
asked Nov 9 at 14:50
Yılmaz
7019
add a comment |
up vote
0
down vote
favorite
I am running flink standalone cluster that consist of 3 servers. I added "env.java.opts: '-Dmykey=1000'" this settings on my 3 nodes, in this file flink-1.6.0/conf/flink-conf.yaml
When I check java programs, I can see this parameter passed through to my TaskManager and JobManager
/usr/lib/jvm/java-8-oracle/bin/java -Xms15360M -Xmx15360M -XX:MaxDirectMemorySize=8388607T -Dmykey=1000 -Dlog.file=/home/kafka/flink-1.6.0/log/flink-
When I try to print out all properties, iterating System.getProperties(), I cannot fetch my custom properties.
I have libraries that uses property files to get external information, I cannot change my libraries, make them use ParameterTools. I did a lot things, but still could not solve this. Is there any advice for me?
apache-flink flink-streaming
asked Nov 9 at 14:50
Yılmaz
7019
You can try querying the configurations for keysenv.java.opts.jobmanagerandenv.java.opts.taskmanager. Refer ci.apache.org/projects/flink/flink-docs-stable/ops/…
– shriyog
Nov 12 at 6:46
@shriyog when I look at the linux processes, I see -Dmykey=1000. But I cannot get it in my program.
– Yılmaz
Nov 13 at 10:27
Did you use the above mentioned keynames for getting values from the configuration map?
– shriyog
Nov 13 at 10:28
@shriyog When I submit my fat jar, I print out all System.getProperties() values. For example, I can see log.file key and value of it, but I cannot see mykey and value of it.
– Yılmaz
Nov 13 at 10:35
I tried it out and it seemed to work for me. Where exactly are you printing the system properties? Please make sure that this happens on the cluster (e.g. in a user code function). If you print it outside of a user function, it will actually be executed on the client.
– Till Rohrmann
Nov 20 at 16:42
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am running flink standalone cluster that consist of 3 servers. I added "env.java.opts: '-Dmykey=1000'" this settings on my 3 nodes, in this file flink-1.6.0/conf/flink-conf.yaml
When I check java programs, I can see this parameter passed through to my TaskManager and JobManager
/usr/lib/jvm/java-8-oracle/bin/java -Xms15360M -Xmx15360M -XX:MaxDirectMemorySize=8388607T -Dmykey=1000 -Dlog.file=/home/kafka/flink-1.6.0/log/flink-
When I try to print out all properties, iterating System.getProperties(), I cannot fetch my custom properties.
I have libraries that uses property files to get external information, I cannot change my libraries, make them use ParameterTools. I did a lot things, but still could not solve this. Is there any advice for me?
apache-flink flink-streaming
asked Nov 9 at 14:50
Yılmaz
7019
I am running flink standalone cluster that consist of 3 servers. I added "env.java.opts: '-Dmykey=1000'" this settings on my 3 nodes, in this file flink-1.6.0/conf/flink-conf.yaml
When I check java programs, I can see this parameter passed through to my TaskManager and JobManager
/usr/lib/jvm/java-8-oracle/bin/java -Xms15360M -Xmx15360M -XX:MaxDirectMemorySize=8388607T -Dmykey=1000 -Dlog.file=/home/kafka/flink-1.6.0/log/flink-
When I try to print out all properties, iterating System.getProperties(), I cannot fetch my custom properties.
I have libraries that uses property files to get external information, I cannot change my libraries, make them use ParameterTools. I did a lot things, but still could not solve this. Is there any advice for me?
apache-flink flink-streaming
apache-flink flink-streaming
asked Nov 9 at 14:50
Yılmaz
7019
asked Nov 9 at 14:50
Yılmaz
7019
asked Nov 9 at 14:50
Yılmaz
7019
asked Nov 9 at 14:50
Yılmaz
7019
asked Nov 9 at 14:50
Yılmaz
7019
7019
You can try querying the configurations for keysenv.java.opts.jobmanagerandenv.java.opts.taskmanager. Refer ci.apache.org/projects/flink/flink-docs-stable/ops/…
– shriyog
Nov 12 at 6:46
@shriyog when I look at the linux processes, I see -Dmykey=1000. But I cannot get it in my program.
– Yılmaz
Nov 13 at 10:27
Did you use the above mentioned keynames for getting values from the configuration map?
– shriyog
Nov 13 at 10:28
@shriyog When I submit my fat jar, I print out all System.getProperties() values. For example, I can see log.file key and value of it, but I cannot see mykey and value of it.
– Yılmaz
Nov 13 at 10:35
I tried it out and it seemed to work for me. Where exactly are you printing the system properties? Please make sure that this happens on the cluster (e.g. in a user code function). If you print it outside of a user function, it will actually be executed on the client.
– Till Rohrmann
Nov 20 at 16:42
add a comment |
You can try querying the configurations for keysenv.java.opts.jobmanagerandenv.java.opts.taskmanager. Refer ci.apache.org/projects/flink/flink-docs-stable/ops/…
– shriyog
Nov 12 at 6:46
@shriyog when I look at the linux processes, I see -Dmykey=1000. But I cannot get it in my program.
– Yılmaz
Nov 13 at 10:27
Did you use the above mentioned keynames for getting values from the configuration map?
– shriyog
Nov 13 at 10:28
@shriyog When I submit my fat jar, I print out all System.getProperties() values. For example, I can see log.file key and value of it, but I cannot see mykey and value of it.
– Yılmaz
Nov 13 at 10:35
I tried it out and it seemed to work for me. Where exactly are you printing the system properties? Please make sure that this happens on the cluster (e.g. in a user code function). If you print it outside of a user function, it will actually be executed on the client.
– Till Rohrmann
Nov 20 at 16:42
You can try querying the configurations for keys
env.java.opts.jobmanager and env.java.opts.taskmanager. Refer ci.apache.org/projects/flink/flink-docs-stable/ops/…– shriyog
Nov 12 at 6:46
You can try querying the configurations for keys
env.java.opts.jobmanager and env.java.opts.taskmanager. Refer ci.apache.org/projects/flink/flink-docs-stable/ops/…– shriyog
Nov 12 at 6:46
@shriyog when I look at the linux processes, I see -Dmykey=1000. But I cannot get it in my program.
– Yılmaz
Nov 13 at 10:27
@shriyog when I look at the linux processes, I see -Dmykey=1000. But I cannot get it in my program.
– Yılmaz
Nov 13 at 10:27
Did you use the above mentioned keynames for getting values from the configuration map?
– shriyog
Nov 13 at 10:28
Did you use the above mentioned keynames for getting values from the configuration map?
– shriyog
Nov 13 at 10:28
@shriyog When I submit my fat jar, I print out all System.getProperties() values. For example, I can see log.file key and value of it, but I cannot see mykey and value of it.
– Yılmaz
Nov 13 at 10:35
@shriyog When I submit my fat jar, I print out all System.getProperties() values. For example, I can see log.file key and value of it, but I cannot see mykey and value of it.
– Yılmaz
Nov 13 at 10:35
I tried it out and it seemed to work for me. Where exactly are you printing the system properties? Please make sure that this happens on the cluster (e.g. in a user code function). If you print it outside of a user function, it will actually be executed on the client.
– Till Rohrmann
Nov 20 at 16:42
I tried it out and it seemed to work for me. Where exactly are you printing the system properties? Please make sure that this happens on the cluster (e.g. in a user code function). If you print it outside of a user function, it will actually be executed on the client.
– Till Rohrmann
Nov 20 at 16:42
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53228032%2fhow-to-get-system-getproperties-in-apache-flink-jobs%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You can try querying the configurations for keys
env.java.opts.jobmanagerandenv.java.opts.taskmanager. Refer ci.apache.org/projects/flink/flink-docs-stable/ops/…– shriyog
Nov 12 at 6:46
@shriyog when I look at the linux processes, I see -Dmykey=1000. But I cannot get it in my program.
– Yılmaz
Nov 13 at 10:27
Did you use the above mentioned keynames for getting values from the configuration map?
– shriyog
Nov 13 at 10:28
@shriyog When I submit my fat jar, I print out all System.getProperties() values. For example, I can see log.file key and value of it, but I cannot see mykey and value of it.
– Yılmaz
Nov 13 at 10:35
I tried it out and it seemed to work for me. Where exactly are you printing the system properties? Please make sure that this happens on the cluster (e.g. in a user code function). If you print it outside of a user function, it will actually be executed on the client.
– Till Rohrmann
Nov 20 at 16:42