IBM Data and AI

Welcome to the IBM Data and AI Ideas Portal for Clients!

We welcome and appreciate your feedback on IBM Data and AI Products to help make them even better than they are today!
Before you submit an idea, please perform a search first as a similar idea may have already been reported in the portal. If a related idea is not yet listed, please create a new idea and include with it a description which includes expected behavior as well as why having this feature would improve the service and how it would address your use case.
IBM Employees:
  • Our team welcomes any feedback and suggestions you have for improving our offerings / products! This forum allows us to connect your offering / product improvement ideas with IBM product and engineering teams.

  • If you have not registered on this portal please click on the following link and register. To complete registration you will need to open the email you will receive from Aha to confirm your identity.

Additional Information:
  • The shorter URL for this site is:

  • To view our roadmaps:

  • Reminder: This is not the place to submit defects or support needs, please use normal support channel for these cases

  • Please do not use the Ideas Portal for reporting bugs - we ask that you report bugs or issues with the product by contacting IBM support.

IAE integartion with SysDig

Dear Team,

We have an Analytics Engine running in our PROD account. This IAE is key to our environment and holds the all the primary jobs being run.

Our ask is how can we collect below list of Spark / Yarn metrics, with the help of monitoring tool Sysdig??

Spark metrics -


Number of jobs
Shown as task


Number of tasks in the application
Shown as task


Number of active tasks in the application
Shown as task


Number of skipped tasks in the application
Shown as task


Number of failed tasks in the application
Shown as task


Number of completed tasks in the application
Shown as task

Yarn cluster metrics



Metric Type


Number of unhealthy nodes

Resource: Error


Number of currently active nodes

Resource: Availability


Number of lost nodes

Resource: Error


Number of failed applications

Work: Error


Total amount of memory/amount of memory allocated

Resource: Utilization


Vishvesh Jain

SRE @ Truata

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Apr 8 2020
  • Planned for future release
Who would benefit from this IDEA? This integration would help us monitor the Analytics engine and get status of jobs being run on Spark/Yarn
Idea Priority Urgent
  • Attach files
  • Admin
    KIRAN Guduguntla commented
    27 Aug 01:35pm

    Hi Vishvesh

    The current IAE plans based on HDP, integrate with New Relic for internal monitoring and metrics. We do not have plans to integrate the current HDP based packages with Sysdig. The new serverless Spark plan that we are working on will integrate with Sysdig. Tentative GA in Frankfurt is Q1'20

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    4 Jun 04:24pm

    Hi Team,

    Hope you guys are doing well!

    Touching base on this one to see if there's a line of sight by when can we expect this feature provisioned in IBM Cloud


    Vishvesh Jain

    SRE @ Truata

NOTICE TO EU RESIDENTS: per EU Data Protection Policy, if you wish to remove your personal information from the IBM ideas portal, please login to the ideas portal using your previously registered information then change your email to "anonymous@euprivacy.out" and first name to "anonymous" and last name to "anonymous". This will ensure that IBM will not send any emails to you about all idea submissions