How to handle ssl / force ssl_version when reading documents with Nokogiri?
up vote
0
down vote
favorite
I have some code that loads a web document using nokogiri:
require 'nokogiri'
require 'open-uri'
require 'openssl'
require 'net/https'
define_method (:loadWebDoc) { |url|
web_doc = nil
begin
file = open(url)
web_doc = Nokogiri::HTML(file)
rescue OpenURI::HTTPError => ex
raise ex
end
web_doc
}
#process some urls with threads...
It's always worked well, until I started using it in threads. My script calls loadWebDoc many times successfully, but after about 30 seconds of processing documents, I get an error like this:
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server session ticket A (OpenSSL::SSL::SSLError)
Here is a similar issue on stack which suggests to use TLSv1, but it's using the stock http and not Nokogiri.
I've tried several variations of something like:file = open(url, :ssl_version => OpenSSL::SSL::SSLContext::TLSv1)
but this just gives me errors likeuninitialized constant OpenSSL::SSL::SSLContext::TLSv1 (NameError)
How can I force Nokogiri to do the same thing? It looks like I need to configure the ssl version and cipher(s) but I'm not sure how with Nokogiri and I'm likely using the wrong constant.
ruby ssl nokogiri
asked Nov 10 at 5:04
kraftydevil
2,50052945
add a comment |
up vote
0
down vote
favorite
I have some code that loads a web document using nokogiri:
require 'nokogiri'
require 'open-uri'
require 'openssl'
require 'net/https'
define_method (:loadWebDoc) { |url|
web_doc = nil
begin
file = open(url)
web_doc = Nokogiri::HTML(file)
rescue OpenURI::HTTPError => ex
raise ex
end
web_doc
}
#process some urls with threads...
It's always worked well, until I started using it in threads. My script calls loadWebDoc many times successfully, but after about 30 seconds of processing documents, I get an error like this:
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server session ticket A (OpenSSL::SSL::SSLError)
Here is a similar issue on stack which suggests to use TLSv1, but it's using the stock http and not Nokogiri.
I've tried several variations of something like:file = open(url, :ssl_version => OpenSSL::SSL::SSLContext::TLSv1)
but this just gives me errors likeuninitialized constant OpenSSL::SSL::SSLContext::TLSv1 (NameError)
How can I force Nokogiri to do the same thing? It looks like I need to configure the ssl version and cipher(s) but I'm not sure how with Nokogiri and I'm likely using the wrong constant.
ruby ssl nokogiri
asked Nov 10 at 5:04
kraftydevil
2,50052945
1
This issue has nothing to do with Nokogiri. Try using rest-client or mechanize and you probably want to set some headers.
– pguardiario
Nov 10 at 7:09
ok so not Nokogiri - do you have some equivalent code that should prevent this connect error?
– kraftydevil
Nov 10 at 9:52
1
Just catch those errors and retry. And maybe use fewer threads.
– pguardiario
Nov 11 at 0:09
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have some code that loads a web document using nokogiri:
require 'nokogiri'
require 'open-uri'
require 'openssl'
require 'net/https'
define_method (:loadWebDoc) { |url|
web_doc = nil
begin
file = open(url)
web_doc = Nokogiri::HTML(file)
rescue OpenURI::HTTPError => ex
raise ex
end
web_doc
}
#process some urls with threads...
It's always worked well, until I started using it in threads. My script calls loadWebDoc many times successfully, but after about 30 seconds of processing documents, I get an error like this:
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server session ticket A (OpenSSL::SSL::SSLError)
Here is a similar issue on stack which suggests to use TLSv1, but it's using the stock http and not Nokogiri.
I've tried several variations of something like:file = open(url, :ssl_version => OpenSSL::SSL::SSLContext::TLSv1)
but this just gives me errors likeuninitialized constant OpenSSL::SSL::SSLContext::TLSv1 (NameError)
How can I force Nokogiri to do the same thing? It looks like I need to configure the ssl version and cipher(s) but I'm not sure how with Nokogiri and I'm likely using the wrong constant.
ruby ssl nokogiri
asked Nov 10 at 5:04
kraftydevil
2,50052945
I have some code that loads a web document using nokogiri:
require 'nokogiri'
require 'open-uri'
require 'openssl'
require 'net/https'
define_method (:loadWebDoc) { |url|
web_doc = nil
begin
file = open(url)
web_doc = Nokogiri::HTML(file)
rescue OpenURI::HTTPError => ex
raise ex
end
web_doc
}
#process some urls with threads...
It's always worked well, until I started using it in threads. My script calls loadWebDoc many times successfully, but after about 30 seconds of processing documents, I get an error like this:
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server session ticket A (OpenSSL::SSL::SSLError)
Here is a similar issue on stack which suggests to use TLSv1, but it's using the stock http and not Nokogiri.
I've tried several variations of something like:file = open(url, :ssl_version => OpenSSL::SSL::SSLContext::TLSv1)
but this just gives me errors likeuninitialized constant OpenSSL::SSL::SSLContext::TLSv1 (NameError)
How can I force Nokogiri to do the same thing? It looks like I need to configure the ssl version and cipher(s) but I'm not sure how with Nokogiri and I'm likely using the wrong constant.
ruby ssl nokogiri
ruby ssl nokogiri
asked Nov 10 at 5:04
kraftydevil
2,50052945
asked Nov 10 at 5:04
kraftydevil
2,50052945
asked Nov 10 at 5:04
kraftydevil
2,50052945
asked Nov 10 at 5:04
kraftydevil
2,50052945
asked Nov 10 at 5:04
kraftydevil
2,50052945
2,50052945
1
This issue has nothing to do with Nokogiri. Try using rest-client or mechanize and you probably want to set some headers.
– pguardiario
Nov 10 at 7:09
ok so not Nokogiri - do you have some equivalent code that should prevent this connect error?
– kraftydevil
Nov 10 at 9:52
1
Just catch those errors and retry. And maybe use fewer threads.
– pguardiario
Nov 11 at 0:09
add a comment |
1
This issue has nothing to do with Nokogiri. Try using rest-client or mechanize and you probably want to set some headers.
– pguardiario
Nov 10 at 7:09
ok so not Nokogiri - do you have some equivalent code that should prevent this connect error?
– kraftydevil
Nov 10 at 9:52
1
Just catch those errors and retry. And maybe use fewer threads.
– pguardiario
Nov 11 at 0:09
1
1
This issue has nothing to do with Nokogiri. Try using rest-client or mechanize and you probably want to set some headers.
– pguardiario
Nov 10 at 7:09
This issue has nothing to do with Nokogiri. Try using rest-client or mechanize and you probably want to set some headers.
– pguardiario
Nov 10 at 7:09
ok so not Nokogiri - do you have some equivalent code that should prevent this connect error?
– kraftydevil
Nov 10 at 9:52
ok so not Nokogiri - do you have some equivalent code that should prevent this connect error?
– kraftydevil
Nov 10 at 9:52
1
1
Just catch those errors and retry. And maybe use fewer threads.
– pguardiario
Nov 11 at 0:09
Just catch those errors and retry. And maybe use fewer threads.
– pguardiario
Nov 11 at 0:09
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
Looks like the error 'connect_nonblock' is raised coz the server can't handle the many connections, especially in the case of threads. Try to add the delay between attempts
open(url, open_timeout: 100)
https://ruby-doc.org/stdlib-2.4.0/libdoc/socket/rdoc/Socket.html#method-i-connect_nonblock
answered Nov 10 at 8:27
Alexey Strizhak
596
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53236142%2fhow-to-handle-ssl-force-ssl-version-when-reading-documents-with-nokogiri%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
Looks like the error 'connect_nonblock' is raised coz the server can't handle the many connections, especially in the case of threads. Try to add the delay between attempts
open(url, open_timeout: 100)
https://ruby-doc.org/stdlib-2.4.0/libdoc/socket/rdoc/Socket.html#method-i-connect_nonblock
answered Nov 10 at 8:27
Alexey Strizhak
596
add a comment |
up vote
1
down vote
Looks like the error 'connect_nonblock' is raised coz the server can't handle the many connections, especially in the case of threads. Try to add the delay between attempts
open(url, open_timeout: 100)
https://ruby-doc.org/stdlib-2.4.0/libdoc/socket/rdoc/Socket.html#method-i-connect_nonblock
answered Nov 10 at 8:27
Alexey Strizhak
596
add a comment |
up vote
1
down vote
up vote
1
down vote
Looks like the error 'connect_nonblock' is raised coz the server can't handle the many connections, especially in the case of threads. Try to add the delay between attempts
open(url, open_timeout: 100)
https://ruby-doc.org/stdlib-2.4.0/libdoc/socket/rdoc/Socket.html#method-i-connect_nonblock
answered Nov 10 at 8:27
Alexey Strizhak
596
Looks like the error 'connect_nonblock' is raised coz the server can't handle the many connections, especially in the case of threads. Try to add the delay between attempts
open(url, open_timeout: 100)
https://ruby-doc.org/stdlib-2.4.0/libdoc/socket/rdoc/Socket.html#method-i-connect_nonblock
answered Nov 10 at 8:27
Alexey Strizhak
596
edited Nov 10 at 9:03
answered Nov 10 at 8:27
Alexey Strizhak
596
answered Nov 10 at 8:27
Alexey Strizhak
596
answered Nov 10 at 8:27
Alexey Strizhak
596
596
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53236142%2fhow-to-handle-ssl-force-ssl-version-when-reading-documents-with-nokogiri%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
This issue has nothing to do with Nokogiri. Try using rest-client or mechanize and you probably want to set some headers.
– pguardiario
Nov 10 at 7:09
ok so not Nokogiri - do you have some equivalent code that should prevent this connect error?
– kraftydevil
Nov 10 at 9:52
1
Just catch those errors and retry. And maybe use fewer threads.
– pguardiario
Nov 11 at 0:09