Exception Java heap space
up vote
0
down vote
favorite
This is my code:
public static void runSGD(double R, double theta, double convergenceTol)
{
List<Integer> allEdges = new ArrayList<Integer>(2*E);
for (int i = 0; i < 2*E; i++)
allEdges.add(i);
Collections.shuffle(allEdges, new Random(shuffleSeed));
double oldRes = calcObj(R, theta, allEdges), newRes = 0.0;
long numEdges = 0;
for (int _e = 0; _e < 2*E*tp; _e++) {
int e = allEdges.get(_e);
numEdges += weights.get(e);
}
if (verbose)
System.out.printf("[Info] Number of edges in training, including multiplicity = %dn", numEdges);
int edgeTable = new int[4][1<<30];
long part = 0; int cur = 0;
for (long i = 0; i < numEdges; i++) {
if (i+1 > part) {
part += weights.get(allEdges.get(cur));
cur++;
}
int row = (int) (i >>> 30);
int col = (int) (i & ((1 << 30) -1));
edgeTable[row][col] = allEdges.get(cur-1);
}
}
The error is Exception in thread "main" java.lang.OutOfMemoryError: Java heap space when run this code:
int edgeTable = new int[4][1<<30];
I have try -Xmx1g,-Xmx3g, but didn't work ,how to fix it?
java
edited Nov 10 at 11:20
AS Mackay
1,8073816
asked Nov 10 at 9:38
Z Mario
246
add a comment |
up vote
0
down vote
favorite
This is my code:
public static void runSGD(double R, double theta, double convergenceTol)
{
List<Integer> allEdges = new ArrayList<Integer>(2*E);
for (int i = 0; i < 2*E; i++)
allEdges.add(i);
Collections.shuffle(allEdges, new Random(shuffleSeed));
double oldRes = calcObj(R, theta, allEdges), newRes = 0.0;
long numEdges = 0;
for (int _e = 0; _e < 2*E*tp; _e++) {
int e = allEdges.get(_e);
numEdges += weights.get(e);
}
if (verbose)
System.out.printf("[Info] Number of edges in training, including multiplicity = %dn", numEdges);
int edgeTable = new int[4][1<<30];
long part = 0; int cur = 0;
for (long i = 0; i < numEdges; i++) {
if (i+1 > part) {
part += weights.get(allEdges.get(cur));
cur++;
}
int row = (int) (i >>> 30);
int col = (int) (i & ((1 << 30) -1));
edgeTable[row][col] = allEdges.get(cur-1);
}
}
The error is Exception in thread "main" java.lang.OutOfMemoryError: Java heap space when run this code:
int edgeTable = new int[4][1<<30];
I have try -Xmx1g,-Xmx3g, but didn't work ,how to fix it?
java
edited Nov 10 at 11:20
AS Mackay
1,8073816
asked Nov 10 at 9:38
Z Mario
246
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
This is my code:
public static void runSGD(double R, double theta, double convergenceTol)
{
List<Integer> allEdges = new ArrayList<Integer>(2*E);
for (int i = 0; i < 2*E; i++)
allEdges.add(i);
Collections.shuffle(allEdges, new Random(shuffleSeed));
double oldRes = calcObj(R, theta, allEdges), newRes = 0.0;
long numEdges = 0;
for (int _e = 0; _e < 2*E*tp; _e++) {
int e = allEdges.get(_e);
numEdges += weights.get(e);
}
if (verbose)
System.out.printf("[Info] Number of edges in training, including multiplicity = %dn", numEdges);
int edgeTable = new int[4][1<<30];
long part = 0; int cur = 0;
for (long i = 0; i < numEdges; i++) {
if (i+1 > part) {
part += weights.get(allEdges.get(cur));
cur++;
}
int row = (int) (i >>> 30);
int col = (int) (i & ((1 << 30) -1));
edgeTable[row][col] = allEdges.get(cur-1);
}
}
The error is Exception in thread "main" java.lang.OutOfMemoryError: Java heap space when run this code:
int edgeTable = new int[4][1<<30];
I have try -Xmx1g,-Xmx3g, but didn't work ,how to fix it?
java
edited Nov 10 at 11:20
AS Mackay
1,8073816
asked Nov 10 at 9:38
Z Mario
246
This is my code:
public static void runSGD(double R, double theta, double convergenceTol)
{
List<Integer> allEdges = new ArrayList<Integer>(2*E);
for (int i = 0; i < 2*E; i++)
allEdges.add(i);
Collections.shuffle(allEdges, new Random(shuffleSeed));
double oldRes = calcObj(R, theta, allEdges), newRes = 0.0;
long numEdges = 0;
for (int _e = 0; _e < 2*E*tp; _e++) {
int e = allEdges.get(_e);
numEdges += weights.get(e);
}
if (verbose)
System.out.printf("[Info] Number of edges in training, including multiplicity = %dn", numEdges);
int edgeTable = new int[4][1<<30];
long part = 0; int cur = 0;
for (long i = 0; i < numEdges; i++) {
if (i+1 > part) {
part += weights.get(allEdges.get(cur));
cur++;
}
int row = (int) (i >>> 30);
int col = (int) (i & ((1 << 30) -1));
edgeTable[row][col] = allEdges.get(cur-1);
}
}
The error is Exception in thread "main" java.lang.OutOfMemoryError: Java heap space when run this code:
int edgeTable = new int[4][1<<30];
I have try -Xmx1g,-Xmx3g, but didn't work ,how to fix it?
java
java
edited Nov 10 at 11:20
AS Mackay
1,8073816
asked Nov 10 at 9:38
Z Mario
246
edited Nov 10 at 11:20
AS Mackay
1,8073816
asked Nov 10 at 9:38
Z Mario
246
edited Nov 10 at 11:20
AS Mackay
1,8073816
edited Nov 10 at 11:20
AS Mackay
1,8073816
edited Nov 10 at 11:20
AS Mackay
1,8073816
1,8073816
asked Nov 10 at 9:38
Z Mario
246
asked Nov 10 at 9:38
Z Mario
246
asked Nov 10 at 9:38
Z Mario
246
246
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
up vote
2
down vote
accepted
You are allocating 4 int arrays of 2^30 ints. That is 2^34 bytes or 16 gigabytes. Clearly that won't fit into a 1 or 3 gigabyte heap. Indeed, a typical laptop or PC won't have enough RAM for this ...
There is a secondary problem about whether the heap spaces are large enough to hold a 2^32 byte object, but it should be possible to address that if you can make the heap large enough.
In fact the JVM supports arrays of just under 2^31 elements; see Do Java arrays have a maximum size?, so the array size per se is not the problem here.
answered Nov 10 at 9:58
Stephen C
511k69560911
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1
4intarrays each containing 2^30intvalues. Eachintis 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.
– Stephen C
Nov 10 at 10:04
add a comment |
up vote
0
down vote
Arrays use ints to address single array elements. The maximum int-value is 2^31-1.
You create 4 Arrays of the size 2^30 which means you have 2^32 elements in your array.
Java does simple not support arrays of that size.
You can fix it by making 4 distinct arrays.
Also allocating 3GB with -Xmx3g wont help, since your array alone will need 16GB of RAM.
answered Nov 10 at 9:49
Max7CD
396
add a comment |
up vote
0
down vote
As others mentioned, you are trying to allocate huge arrays. You can try by allocating chunks of small array or try collections.
answered Nov 10 at 12:30
mani deepak
314313
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53237682%2fexception-java-heap-space%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
You are allocating 4 int arrays of 2^30 ints. That is 2^34 bytes or 16 gigabytes. Clearly that won't fit into a 1 or 3 gigabyte heap. Indeed, a typical laptop or PC won't have enough RAM for this ...
There is a secondary problem about whether the heap spaces are large enough to hold a 2^32 byte object, but it should be possible to address that if you can make the heap large enough.
In fact the JVM supports arrays of just under 2^31 elements; see Do Java arrays have a maximum size?, so the array size per se is not the problem here.
answered Nov 10 at 9:58
Stephen C
511k69560911
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1
4intarrays each containing 2^30intvalues. Eachintis 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.
– Stephen C
Nov 10 at 10:04
add a comment |
up vote
2
down vote
accepted
You are allocating 4 int arrays of 2^30 ints. That is 2^34 bytes or 16 gigabytes. Clearly that won't fit into a 1 or 3 gigabyte heap. Indeed, a typical laptop or PC won't have enough RAM for this ...
There is a secondary problem about whether the heap spaces are large enough to hold a 2^32 byte object, but it should be possible to address that if you can make the heap large enough.
In fact the JVM supports arrays of just under 2^31 elements; see Do Java arrays have a maximum size?, so the array size per se is not the problem here.
answered Nov 10 at 9:58
Stephen C
511k69560911
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1
4intarrays each containing 2^30intvalues. Eachintis 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.
– Stephen C
Nov 10 at 10:04
add a comment |
up vote
2
down vote
accepted
up vote
2
down vote
accepted
You are allocating 4 int arrays of 2^30 ints. That is 2^34 bytes or 16 gigabytes. Clearly that won't fit into a 1 or 3 gigabyte heap. Indeed, a typical laptop or PC won't have enough RAM for this ...
There is a secondary problem about whether the heap spaces are large enough to hold a 2^32 byte object, but it should be possible to address that if you can make the heap large enough.
In fact the JVM supports arrays of just under 2^31 elements; see Do Java arrays have a maximum size?, so the array size per se is not the problem here.
answered Nov 10 at 9:58
Stephen C
511k69560911
You are allocating 4 int arrays of 2^30 ints. That is 2^34 bytes or 16 gigabytes. Clearly that won't fit into a 1 or 3 gigabyte heap. Indeed, a typical laptop or PC won't have enough RAM for this ...
There is a secondary problem about whether the heap spaces are large enough to hold a 2^32 byte object, but it should be possible to address that if you can make the heap large enough.
In fact the JVM supports arrays of just under 2^31 elements; see Do Java arrays have a maximum size?, so the array size per se is not the problem here.
answered Nov 10 at 9:58
Stephen C
511k69560911
edited Nov 11 at 7:32
answered Nov 10 at 9:58
Stephen C
511k69560911
answered Nov 10 at 9:58
Stephen C
511k69560911
answered Nov 10 at 9:58
Stephen C
511k69560911
511k69560911
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1
4intarrays each containing 2^30intvalues. Eachintis 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.
– Stephen C
Nov 10 at 10:04
add a comment |
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1
4intarrays each containing 2^30intvalues. Eachintis 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.
– Stephen C
Nov 10 at 10:04
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1 int 4 = 2^2 bytes 2^30 int 2^32 bytes how come 2^34 bytes could you explain?
– snr
Nov 10 at 10:02
1
1
4
int arrays each containing 2^30 int values. Each int is 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.– Stephen C
Nov 10 at 10:04
4
int arrays each containing 2^30 int values. Each int is 4 bytes. 4 x 4 x 2^30 == 16 x 2^30 == 16 gigabytes.– Stephen C
Nov 10 at 10:04
add a comment |
up vote
0
down vote
Arrays use ints to address single array elements. The maximum int-value is 2^31-1.
You create 4 Arrays of the size 2^30 which means you have 2^32 elements in your array.
Java does simple not support arrays of that size.
You can fix it by making 4 distinct arrays.
Also allocating 3GB with -Xmx3g wont help, since your array alone will need 16GB of RAM.
answered Nov 10 at 9:49
Max7CD
396
add a comment |
up vote
0
down vote
Arrays use ints to address single array elements. The maximum int-value is 2^31-1.
You create 4 Arrays of the size 2^30 which means you have 2^32 elements in your array.
Java does simple not support arrays of that size.
You can fix it by making 4 distinct arrays.
Also allocating 3GB with -Xmx3g wont help, since your array alone will need 16GB of RAM.
answered Nov 10 at 9:49
Max7CD
396
add a comment |
up vote
0
down vote
up vote
0
down vote
Arrays use ints to address single array elements. The maximum int-value is 2^31-1.
You create 4 Arrays of the size 2^30 which means you have 2^32 elements in your array.
Java does simple not support arrays of that size.
You can fix it by making 4 distinct arrays.
Also allocating 3GB with -Xmx3g wont help, since your array alone will need 16GB of RAM.
answered Nov 10 at 9:49
Max7CD
396
Arrays use ints to address single array elements. The maximum int-value is 2^31-1.
You create 4 Arrays of the size 2^30 which means you have 2^32 elements in your array.
Java does simple not support arrays of that size.
You can fix it by making 4 distinct arrays.
Also allocating 3GB with -Xmx3g wont help, since your array alone will need 16GB of RAM.
answered Nov 10 at 9:49
Max7CD
396
edited Nov 10 at 10:12
answered Nov 10 at 9:49
Max7CD
396
answered Nov 10 at 9:49
Max7CD
396
answered Nov 10 at 9:49
Max7CD
396
396
add a comment |
add a comment |
up vote
0
down vote
As others mentioned, you are trying to allocate huge arrays. You can try by allocating chunks of small array or try collections.
answered Nov 10 at 12:30
mani deepak
314313
add a comment |
up vote
0
down vote
As others mentioned, you are trying to allocate huge arrays. You can try by allocating chunks of small array or try collections.
answered Nov 10 at 12:30
mani deepak
314313
add a comment |
up vote
0
down vote
up vote
0
down vote
As others mentioned, you are trying to allocate huge arrays. You can try by allocating chunks of small array or try collections.
answered Nov 10 at 12:30
mani deepak
314313
As others mentioned, you are trying to allocate huge arrays. You can try by allocating chunks of small array or try collections.
answered Nov 10 at 12:30
mani deepak
314313
answered Nov 10 at 12:30
mani deepak
314313
answered Nov 10 at 12:30
mani deepak
314313
answered Nov 10 at 12:30
mani deepak
314313
314313
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53237682%2fexception-java-heap-space%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
