Working with Big Data databases, the enormous volume of which makes it difficult to use common data analysis methods, requires the use of advanced statistical tools to perform data quality control and draw insights from information.
The Bank of Israel’s Information and Statistics Department manages a database of Big Data on derivative financial instruments in the forex and interest rate markets, which contains information on transactions in these derivatives on the OTC market. The database is used primarily to gain knowledge of the market and to develop products that support the decisions of the Bank’s policy makers.
This study reviews several tools and methods that the Department uses to optimize, process, and generate insights from the information existing in the derivatives database. The study also describes several data science models that are used to identify irregular reports, complete missing data, classify market participants, and enable the Department to draw insights that inform policy making.