Step performance graphs

One of the things I’ve been working on lately in Kettle / Pentaho Data Integration is the transparency of the performance monitoring.

We don’t just need an API to get the step performance data out, but we also need to visualize this data in a simple way, something like this:
performance graph

Graph with moving average

The next steps will be to also allow this data to be spooled off to a database somewhere and to be accessed remotely using Carte.

Until next time,

Matt

9 thoughts on “Step performance graphs”

  1. Hé Matt..that’s great !
    I see that Santa clause visited you 🙂
    Is this step will be available in 3.1 (as a plugin or in the code base).

    Any news from data quality plugin?

    A+

    Samatar

  2. Hi Samatar,

    It’s no step, it will be available directly from the transformation log view in v3.1 as well as the slave server logging view. (remote)

    No news from the data quality plugin, but I have plans of my own 😉

    Cheers,

    Matt

  3. It’s no step, it will be available directly from the transformation log view in v3.1 as well as the slave server logging view. (remote)
    –> Super

    No news from the data quality plugin, but I have plans of my own
    –> with 3.1 of after?

    Take care

  4. That’s great Matt..
    Adding this feature will bring more help on easily point on bottleneck steps.
    Nice…Really Nice.

    Samatar

  5. Fantabulous (fantastic + fabulous)…

    >> The next steps will be to also allow this data to be spooled off to a database somewhere.

    Will this be available in 3.1 ??

    Can we help ??

    -rusang

  6. rusang, 3.1.0-M1 will indeed carry these changes. You can download one of the nightly builds to try:

    ftp://download.pentaho.org/client/pentaho-data-integration/3.1.0

    To write to disk, I’m thinking about making a step available that exposes the performance data from the result object. Pentaho HQ in Orlando is also creating a management console that will create fancy reports and analytics on that data, probably later next month or so.

    Anyone can help!! If you have a good idea, I would advice you to create a JIRA case or send an e-mail to the kettle-developers mailing list (@ google groups).

    Cheers,

    Matt

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.