how to save database versioning as a sql_last_version variable in logstash
The parameter is called sql_last_value and here is how you can use it:
input { jdbc { jdbc_connection_string => "jdbc:postgresql://postgres:5432/test" jdbc_driver_library => "/home/postgresql-42.2.18.jar" # last_run_metadata_path => "/usr/share/logstash/.logstash_jdbc_last_run" jdbc_driver_class => "org.postgresql.Driver" jdbc_user => "postgres" jdbc_password => "postgres" schedule => "*/5 * * * * *" use_column_value => true tracking_column => "xmin" statement => "SELECT * FROM test where xmin::text::int >= :sql_last_value ORDER BY xmin::text::int " }}Here are the modification I made:
- You don't need the
parameterssetting - You're missing the
use_column_valuesetting - You're missing the
tracking_columnsetting - The SQL query needs to reference
sql_last_valuenotsql_last_version
After the first run, 600 will be stored in sql_last_value and used in the next call to only retrieve records whose xmin value is >= 600 (which means the last record will be fetched again)