Using Infosphere Information Server 11.7.1. Service Pack 2, we noticed that we are unable to create a data connection for Amazon S3 in DataStage to a private endpoint (not the usual public Amazon endpoints).
We are unable to read/write information through a data stage job from a bucket. The same problem we have encoutered in IMAM also.
IBM Engineering team provided some development jars, and an extra variable was added in Administrator Client, through PMR (Case TS003424088), in order to solve the problem, but an enchancement to the product is nedeed.
On the IMAM side the environent variable was added to dsenv.
|Who would benefit from this IDEA?||ING Data Lakes|
How should it work?
We should be able to establish connection to S3 Buckets (S3A and S3 connectivity) and read/write objects through the already present Datastage Amazon S3 connector. IIS should be able to connect to any endpoint that provides S3/S3A API compliance (private / public).
An option to skip endpoint SSL certificate validation in DEV & TEST is also required.
Also, we should be able to import metadata from S3 Buckets files / objects into IGC using IMAM (osh file needs to be present in the bucket for every file that has to be imported).
|Priority Justification||Introduce the Object Storage technology inside Data Lake|
NOTICE TO EU RESIDENTS: per EU Data Protection Policy, if you wish to remove your personal information from the IBM ideas portal, please login to the ideas portal using your previously registered information then change your email to "email@example.com" and first name to "anonymous" and last name to "anonymous". This will ensure that IBM will not send any emails to you about all idea submissions