Datapipe Files

Real-time, incremental ETL library for ML with record-level depend

Status: Alpha

Brought to you by: otatarinova

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2022-04-29	990 Bytes	0
v0.11.0-beta.1 source code.tar.gz	2022-04-29	758.3 kB	0
v0.11.0-beta.1 source code.zip	2022-04-29	785.9 kB	0
Totals: 3 Items		1.5 MB	0

Несовместимые изменения

Вернулся класс ComputeStep как основа вычислительного пайплайна
Введено понятие full и changelist обработки
batch_transform_wrapper переименован в do_full_batch_transform
Появился метод do_batch_transform который принимает итератор индексов и делает итерационную обработку
batch_generate_wrapper переименован в do_batch_generate
Исключения в do_batch_generate больше не скрываются
По всему коду параметр batchsize переименован в batch_size для консистентности с input_dts, change_list и тп
DataStore.get_process_ids переименован в get_full_process_ids
Добавлен метод get_change_list_process_ids

Source: README.md, updated 2022-04-29

Other Useful Business Software

Build on Google Cloud with $300 in Free Credit Icon

Build on Google Cloud with $300 in Free Credit

New to Google Cloud? Get $300 in free credit to explore Compute Engine, BigQuery, Cloud Run, Vertex AI, and 150+ other products.

Start your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query exabytes in BigQuery, or build AI apps with Vertex AI and Gemini. Once your credits are used, keep building with 20+ products with free monthly usage, including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. Sign up to start building right away.

Start Free Trial

Build AI Apps with Gemini 3 on Vertex AI Icon

Build AI Apps with Gemini 3 on Vertex AI

Access Google’s most capable multimodal models. Train, test, and deploy AI with 200+ foundation models on one platform.

Vertex AI gives developers access to Gemini 3—Google’s most advanced reasoning and coding model—plus 200+ foundation models including Claude, Llama, and Gemma. Build generative AI apps with Vertex AI Studio, customize with fine-tuning, and deploy to production with enterprise-grade MLOps. New customers get $300 in free credits.

Try Vertex AI Free

Recommended Projects

CloverDX
Design, automate, operate and publish data pipelines at scale
Pentaho
Pentaho offers comprehensive data integration and analytics platform.
MixDEM
MixDEM a web based ETL tools meant for Web integration, Data transformation and Mashup edition. It include MixDEM ETL Engine created using ZEND Framework, and MixDEM GUI Editor an AJAX IDE that enable developers to quickly and easily create applications.
rudderstack
Privacy and Security focused Segment-alternative, in Golang
GETL
ETL engine based on Groovy