Open Source Linux ETL Tools - Page 2

P.S. Dear friends. Repository migration to https://github.com/ascrus/getl . You can download jar file from this site or maven. GETL - based package in Groovy, which automates the work of loading and transforming data. His name is an acronym for «Groovy ETL». GETL is a set of libraries of pre-built classes and objects that can be used to solve problems unpacking, transform and load data into programs written in Groovy, or Java, as well as from any software that supports the work with Java classes. GETL taken into account when developing ideas and following requirements: 1. The simpler the class hierarchy, the easier solution; 2. The data structures tend to change over time, or not be known in advance, working with them must be maintained; 3. All routine work ETL should be automated wherever possible; 4. Compiling the code on the fly bail speed and reserve for the optimization; 5. Sophisticated class hierarchy guarantee easy connection of other open source solutions.

The Iguana is a BI solution generation data management and decision making for Open Sources systems used worldwide and Commercial Systems used in Brazil.

Createinfactl is a Java utility that enables Administrators to fully automate Informatica deployments from the command line by creating thedeployment group control XML file to be used with the pmrep command “deploydeploymentgroup”. Default settings for the control file can be overridden at the command line and works with both static and dynamic deployment groups in the repository. Please review the “Using the Deployment Control File” section in the Informatica Command Reference guide for further help on the deployment control file. This utility supports Informatica PowerCenter versions 8.6.1 onwards and Java 1.6\. This utility also contains JDBC drivers for DB2, SQL Server, Oracle, and Teradata databases.

Dbmetadata is a Java utility that reads the metadata from table(s) in a specified database and creates the Informatica XML to import into the repository. I created this utility when we were migrating to a new platform and needed a quick way to create flatfile and relational sources and targets that matched the DDL of the table. I also needed to use shortcuts. If you use the import table list, it will create one XML file with all of the tables and shortcuts (if a shortcut folder is specified) for the requested output type and database/file type.

A simple Java ASN BER decoder and profiler A tool for easy manipulation of BER encoded files. An "awk" for ASN.1 BER (for Unix people) or maybe a "notepad" for ASN.1 BER (for Windows people). Jberd (Java BER decoder) is a lightweight BER decoder and associated tools for interpreting and processing BER encoded ASN.1 files. The following facilities are provided: • JBerd Profiler. A tool for profiling the contents of BER encoded files • JBerd Flattener. A tool for converting BER encoded files to flat files for processing by other facilities • JBerd Decoder objects. A set of Java facilities for writing BER applications that require BER decoding Go to the "files" section (link at the top of this page) to download a pdf of detailed documentation. Andrew Forsyth

Jipes provides open source Java APIs deeply integrated into the Oracle RDBMS, including an Ant task for building and exporting database objects. A Java Data Cartridge replacing database links is also in process.

Simple and easy ETL tool useful for small data warehouse projects. Written in Java.

MixDEM a web based ETL tools meant for Web integration, Data transformation and Mashup edition. It include MixDEM ETL Engine created using ZEND Framework, and MixDEM GUI Editor an AJAX IDE that enable developers to quickly and easily create applications.

Misc scripts and utilities related to Oracle Warehouse Builder ETL (Tcl scripts, OWB Expert, project samples, etc.)

File Based DBMS & ETL Tool: OpenSQL is a file based database management system which uses SQL like features to accept the query request and return the query response. In later phase of this project, ETL based features will be added.

A metadata driven 'tool' to automate loading a designed Data Vault. It consists of a set of Pentaho Data Integration and database objects. Thel Virtual Machine (VMware) is a 64 bit Ubuntu Server 14.04, with MySQL (Percona Server) and PostgreSQL 9.4 as the database flavours and PDI version 5.2 CE. NB: Directory version_2.4 contains the most recent Virtual Machine. The readme.txt contains info about that VM.

PanBI is a collection of analytics modules for existing information systems. For each IS, it provides data extraction, transformation and loading logic coupled with an OLAP schema, delivering OLAP functionality to an unprecedented user base.

ETL Framework is a standalone Extract Transform Load engine written in Java. It includes executables for all major platforms and can be easily integrated into other applications. Key Features: * embeddable, open source and free * fast and scalable * uses target database features to do transformations and loads * manual and automatic data mapping * data streaming * bulk data loads * data quality features using SQL, JavaScript? and regex * data transformations Requirements * Java 1.6 and up * At least 4 MB of RAM New in 3.2 (01/18/2013) * Improved auto-update functionality * Bug fixes

TopGun Twitter Analytics is an open source data warehouse for collecting and analyzing Twitter topics. A topic is made up of one or more keywords.

Compose Transporter helps with database transformations from one store to another. It can also sync from one to another or several stores. This version officially only supports the mongodb and postgresql adaptors. Support for other DBs will be added later on. Other adaptors may or may not work. You're encouraged to still use v0.5.2 for non mongo/postgres migrations. Transporter allows the user to configure a number of data adaptors as sources or sinks. These can be databases, files or other resources. Data is read from the sources, converted into a message format, and then send down to the sink where the message is converted into a writable format for its destination. The user can also create data transformations in JavaScript which can sit between the source and sink and manipulate or filter the message flow. Adaptors may be able to track changes as they happen in source data. This "tail" capability allows a Transporter to stay running and keep the sinks in sync.

XIForge is a team of IT volunteer to explore new free open source technology framework and platform. We focus Pentaho and OpenBravo ERP. Our current hosted project includes Pentaho Data Integration Parse JSON String plugin. Team founder is Reid Lai.

The aDORe Federation is a standards-based federated repository framework and reference implementation which aims to address many of the scalability issues experienced by large scale digital object repositories.

Saas E-commerce Platform

This is an offshoot project of open source data quality (osDQ) project https://sourceforge.net/projects/dataquality/ This sub project will create apache spark based data pipeline where JSON based metadata (file) will be used to run data processing , data pipeline , data quality and data preparation and data modeling features for big data. This uses java API of apache spark. It can run in local mode also. Get json example at https://github.com/arrahtech/osdq-spark How to run Unzip the zip file Windows : java -cp .\lib\*;osdq-spark-0.0.1.jar org.arrah.framework.spark.run.TransformRunner -c .\example\samplerun.json Mac UNIX java -cp ./lib/*:./osdq-spark-0.0.1.jar org.arrah.framework.spark.run.TransformRunner -c ./example/samplerun.json For those on windows, you need to have hadoop distribtion unzipped on local drive and HADOOP_HOME set. Also copy winutils.exe from here into HADOOP_HOME\bin

Better SQL in java! Offering a seamless java class mapping and SQL-like domain-specific language implemented for number of commercial and open-source DBMS

This project is meant to do all the dirty threading work for you. The intention is to use this project as an archetype to provide a framework for writing ETLs. It contains an Extractor Thread, a Loader Thread and a Transformer Thread. All that is needed is to add the necessary business logic for your ETL while not having to worry about making sure your threading is correct.

Diffs, patches, and revision control for CSV files, spreadsheets, and databases.

The cplusql distributed ETL tool extracts and transforms row based data from databases and flat files for terabyte scale datawarehouse loading. RSN: beowulf job queueing and cluster fs! Every day at LookSmart, it translates 200+mm rows on 100+ nodes.

Document Management System created using JEE6