What's the best way to save test data to a database from jenkins/hudson?
Several possibilities:
Use an existing java sql library with Grab
There are several world class java sql libs out there (https://commons.apache.org/proper/commons-dbutils/, https://github.com/jOOQ/jOOQ) which you can use in your pipeline in conjunction with Grab. Personally I love jooq. Then you write your specific code in groovy.
@Grab(group='org.jooq', module='jooq', version='3.10.5')import ...Use an existing java library and put it on the classpath
Like the solution above. But you don't import the library with Grab. Instead starting jenkins from the war directly, you maintain your own jetty installation (https://wiki.jenkins.io/display/JENKINS/Jetty). This provides you the freedom to add additional libraries to the classpath, which can be used then in the pipelines.
Write a java library and use it with Grab or classpath
Instead of using a general purpose library and code your specific requirements in groovy. You create a library, which provides an API for your needs. You import it with Grab or you add it to the classpath of the application server. In your groovy scripts you just have to call the specific API and pass the parameters.
Write your own plugin
You can write a plugin for your purpose which is not so difficult https://jenkins.io/doc/developer/plugin-development/pipeline-integration/. The plugin would take the file path as parameter. The contract will be the structure (xml schema) of the xml file. You could also add parameters to control the import.
Personal opinion:
I would go with "Write a java library and use it with Grab or classpath":
- good testing possibilities on unit level, because of simplicity
- no dependency of the implementation of the specific requirements to jenkins, means can also be used without jenkins
- easy to distribute, can be deployed to any maven compliant repository and fetched from there
@GrabResolver(name='custom', root='http://customserver/repo', m2Compatible='true')
What we are using at the company (and at some other projects I've involved) is the FluentD. You can find a well-maintained Jenkins plugin here or here.
The main features are: sending any valid json, which can be analyzed later using your own tool, something like ELK stack (you can find a lot HOWTO articles), or some other stacks.
Why FluentD? I guess it clearly states on the author's github page: " Our choice was to use Fluentd as looks mature enough to handle a lot of data and it supports a lot of destination endpoints (DBs, file, http, etc)." The data outputs could be anythong you wantL Hadoop, Mongo, S3, AWS/Azure services, or any existing database (https://www.fluentd.org/dataoutputs), and a lot of visualization.
I assume you can ask "why not push it directly to the database?". The answer is - you will need to push it in a specific format and the data won't be easily formatted to another one. I.e. You've built your own tool, but later realized you can use Kibana instead, so you'll need to migrate over. If you start using the FluentD, the only thing you should think about is how to format the json, which is much more trivial, than designing a database. Also, FluentD is highly scalable and supports robust failovers.
One more potential issue is converting XML, you've mentioned you have as test output, to json. First of all, a lot of test runners support json formatting. Secondly, there's a common method of generating json out of xml and I would suggest something like "xml2json" tool, which does a pretty good job:
XML JSON
<e/> "e": null<e>text</e> "e": "text"<e name="value" /> "e": { "@name": "value" }<e name="value">text</e> "e": { "@name": "value", "#text": "text" }<e> <a>text</a ><b>text</b> </e> "e": { "a": "text", "b": "text" }<e> <a>text</a> <a>text</a> </e> "e": { "a": ["text", "text"] }<e> text <a>text</a> </e> "e": { "#text": "text", "a": "text" }