IATI Data Store

New amendment added #1: The tender is hereby amended as follow:- Dead Line for submission has been extended by 2 days, to 7th August 2018 at 23:59 UTC.

New clarification added: Question: IATI Registry - is it safe to say that existing IATI Registry is not undergoing any changes for the new implementation of Data Store? Answer: No changes will take place with regard to the way in which data is accessed. However validation service will be added and the resulting metadata will be available for the datastore to consume Page 3 - Phase -1, under ETL Section:>> Metadata should be maintained that records all validation errors (via Registry) and transformation processes for each activity.ETL validation errors should be reported back to the IATI registry? or should the validation errors be maintained in the Data Store and exposed via API? Answer: The development of a new standardised validation service that is available to all applications and users is under review. The DataStore can either access validation metadata from the Registry or run this standard validation. Users of the Datastore should have access to the validation metadata. In addition the DataStore should maintain its own logs of the ETL process, including any changes made to the data during these processes. Page 3/4 - Phase 2>> Analyst Interface and Publisher Interface --a.       Are there any technology requirements for developing these user interfaces? Web or Desktop tools? Answer: Beyond the overriding preference for Python, no. A desktop tool was not part of this RFP but the benefits of such a service would be considered. b.       If web, does it remain separate or should be integrated to thee main IATI website? Answer: It should sit on its own platform on an iatistandard.org sub-domain, but be seamlessly branded with the main website We found an existing implementation of Data Store on your github, wondering if you can highlight things that you definitely want us to improve (peformance, scalability, relability) or in the new implementation? Answer: Robust and reliable extraction (including accurate handling of deleted activities and datasets that are temporarily unavailable); a database structure that allows proper indexing and filtering of all fields; access to all data in one version of the standard; a sustainable, extensible architecture that makes the handling of upgrades to the standard easy to implement; access to metadata

New clarification added: Question: Dear Team,Please confirm the understanding and answer following. UNOPS already publishes data to IATI and requires the below:Section: Outputs, Page 2:1. A data store that should store all IATI metadata and Activity/Organization’s transaction data fetched from IATI registry using Web APIs?Answer: Yes2. Create a data service accessible to anybody that should query the Data Store and render the results based on parameters/filters passed to it?Answer: Yes3. Data Store should keeping polling IATI registry for new data or metadata and update itself. What frequency you are looking at for these updates/refresh because it certainly does not require real time data streaming?Answer: Data must be refreshed at least once per day, but preferably more frequently.4. Is there an intention to fetch IATI metadata too using data service for IATI publishing?Answer: Yes, the Registry holds metadata on the publisher and on the validation of datasets.5.  How does UNOPS want to store IATI data/metadata in Data Store? In file format or in normalized table schema?Answer: The proposed architecture is part of the proposal.6. Since UNOPS is already IATI compliant, why does it not uses its current data store/data mart/data warehouse for rendering data to public? How UNOPS current data store would be different than the one mentioned/required in this RFP?Answer: The IATI data store is a data service for all data from all publishers to be easily accessible from a single sourceThe IATI data store is a data service for all data from all publishers to be easily accessible from a single source​7. When the data is published in previous version (i.e. 2.01) of IATI standards it is not advisable to transform same data in latest versions (2.03) and render in XML/JSON formats as it may lose its correctness/completeness (due to standard's tag/element changes). Does it still remain a requirement?Answer: Yes. This is an essential requirement to ensure that all data can be compared, irrespective of the version published​ Section: General1. At what frequency UNOPS currently publishes implementation data to IATI?Answer: UNOPS publishes IATI data on a monthly basis​ System Requirements, Page 41. Is it mandatory to develop data service using Python and its code repository should be versioned using Github only? Are you open for other technology stack?Answer: If a long-term sustainable solution is proposed it may be considered. See the previous request for clarification for an explanation

New clarification added: Question: Under the System Requirements > Coding Standards, there is a requirement to use "python 3" and a specific library for XML processing (lxml). Is the python 3 requirement for the overall platform or just for XML processing? If this applies to the overall platform, can organizations propose alternatives, including justification for these alternatives?Answer: The python 3 requirement is for the overall platform. The reason for this is that all core IATI products are written in (or are being converted to) python to standardise the technologies employed by the IATI Secretariat’s Technical Team. An alternative open source codebase could be considered if a very clear case can be made for a long-term, sustainable hosting solution.

New clarification added: The python 3 requirement is for the overall platform. The reason for this is that all core IATI products are written in (or are being converted to) python to standardise the technologies employed by the IATI Secretariat’s Technical Team. An alternative open source codebase could be considered if a very clear case can be made for a long-term, sustainable hosting solution.