Xtykutl

I have problem with jdbc_driver_library.

ELK_VERSION = 6.4.2

And I use Docker for elk.

when i run:

/opt/logstash# bin/logstash -f /etc/logstash/conf.d/mysql.conf

Get Error:

error: com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?

driver path:

root@xxxxxxx:/etc/logstash/conectors# ls

mysql-connector-java-8.0.12.jar 

root@xxxxxxxxxx:/etc/logstash/conectors#

mysql.conf:

input {

  jdbc {

    jdbc_driver_library => "/etc/logstash/conectors/mysql-connector-java-8.0.12.jar"

    jdbc_driver_class => "com.mysql.jdbc.Driver"

    jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"

    jdbc_user => "demouser"

    jdbc_password => "demopassword"

    statement => "SELECT id,name,city from ads"

  }

}



output {

stdout { codec => rubydebug }



     elasticsearch {

        index => 'test'

        document_type => 'tes'

        document_id => '%{id}'

        hosts => ['http://localhost:9200']

     }



 }

Whole error:

root@xxxxx:/opt/logstash# bin/logstash -f /etc/logstash/conf.d/mysql.conf

Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties

[2018-11-10T09:03:22,081][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

[2018-11-10T09:03:23,628][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.2"}

[2018-11-10T09:03:30,482][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2018-11-10T09:03:31,479][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}

[2018-11-10T09:03:31,928][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}

[2018-11-10T09:03:32,067][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}

[2018-11-10T09:03:32,076][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}

[2018-11-10T09:03:32,154][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}

[2018-11-10T09:03:32,210][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

[2018-11-10T09:03:32,267][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

[2018-11-10T09:03:32,760][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x202f727c run>"}

[2018-11-10T09:03:32,980][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}

[2018-11-10T09:03:33,877][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[2018-11-10T09:03:34,315][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.

  Pipeline_id:main

  Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"demouser", jdbc_password=><password>, statement=>"SELECT id,name,city from ads", jdbc_driver_library=>"/etc/logstash/conectors/mysql-connector-java-8.0.12.jar", jdbc_connection_string=>"jdbc:mysql://localhost:3306/mydb", id=>"233c4411c2434e93444c3f59eb9503f3a75cab4f85b0a947d96fa6773dac56cd", jdbc_driver_class=>"com.mysql.jdbc.Driver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_cf5ab80c-91e4-4bc4-8d20-8c5a0f9f8077", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 00:00:00 +0000}, last_run_metadata_path=>"/root/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>

  Error: com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?

  Exception: LogStash::ConfigurationError

  Stack: /opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'

/opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:in `execute_statement'

/opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in `execute_query'

/opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:263:in `run'

/opt/logstash/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'

/opt/logstash/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'

when I build image and docker run

ERROR:

[2018-11-10T10:32:52,935][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/opt/logstash/data/queue"}

[2018-11-10T10:32:52,966][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/opt/logstash/data/dead_letter_queue"}

[2018-11-10T10:32:54,509][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Same problem when i use postgresql

psql.conf

input {

   jdbc {

     type => 'test'

     jdbc_driver_library => '/etc/logstash/postgresql-9.1-901-1.jdbc4.jar'

     jdbc_driver_class => 'org.postgresql.Driver'

     jdbc_connection_string => 'jdbc:postgresql://localhost:5432/mytestdb'

     jdbc_user => 'postgres'

     jdbc_password => 'xxxxxx'

     jdbc_page_size => '50000'

     statement => 'SELECT id, name, city FROM ads'

   }

 }

Then I run

/opt/logstash# bin/logstash -f /etc/logstash/conf.d/psql.conf

ERROR:

error: org.postgresql.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?

In MySQL 8 that you're using, the JDBC driver was renamed from com.mysql.jdbc.Driver to com.mysql.cj.jdbc.Driver (see the release notes for details). Just update your jdbc_driver_class configuration and you should be OK.

I solved the problem:

First check your java version:

root@xxxxxx:/# java -version

openjdk version "1.8.0_181"

If you are using 1.8 then you should use the JDBC42 version.

If you are using 1.7 then you should use the JDBC41 version.

If you are using 1.6 then you should use the JDBC43 version.

postgres setup:

postgresql-9.4-1203.jdbc42.jar

 jdbc_driver_library => '/path_to_jar/postgresql-9.4-1203.jdbc42.jar'

 jdbc_driver_class => 'org.postgresql.Driver'

mysql setup:

mysql-connector-java-5.1.46.jar

 jdbc_driver_library => "//path_to_jar/mysql-connector-java-5.1.46.jar"

 jdbc_driver_class => "com.mysql.jdbc.Driver"