Datapipe Files

Real-time, incremental ETL library for ML with record-level depend

Status: Alpha

Brought to you by: otatarinova

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2023-07-30	572 Bytes	0
v0.13.0-alpha.7 source code.tar.gz	2023-07-30	601.6 kB	0
v0.13.0-alpha.7 source code.zip	2023-07-30	637.4 kB	0
Totals: 3 Items		1.2 MB	0

Try to setup logging in RayExecutor (fails so far)
Lazy initialisation of Ray to speedup things in CLI
Add ExecutorConfig.parallelism parameter
Add name parameter to executor.run_process_batch to customize task name in ray dashboard
Migrate run_changelist to executor, possible parallelisation
Limit number of in-flight Ray tasks in one run_process_batch to 100
Fix batch count in tqdm in run_changelist
Add --start-step parameter to step run-changelist CLI
Move --executor parameter from datapipe step to datapipe command

Source: README.md, updated 2023-07-30

Other Useful Business Software

Run Any Workload on Compute Engine VMs

From dev environments to AI training, choose preset or custom VMs with 1–96 vCPUs and industry-leading 99.95% uptime SLA.

Compute Engine delivers high-performance virtual machines for web apps, databases, containers, and AI workloads. Choose from general-purpose, compute-optimized, or GPU/TPU-accelerated machine types—or build custom VMs to match your exact specs. With live migration and automatic failover, your workloads stay online. New customers get $300 in free credits.

Try Compute Engine

Deploy Apps in Seconds with Cloud Run

Host and run your applications without the need to manage infrastructure. Scales up from and down to zero automatically.

Cloud Run is the fastest way to deploy containerized apps. Push your code in Go, Python, Node.js, Java, or any language and Cloud Run builds and deploys it automatically. Get fast autoscaling, pay only when your code runs, and skip the infrastructure headaches. Two million requests free per month. And new customers get $300 in free credit.

Try Cloud Run Free

Recommended Projects

CloverDX
Design, automate, operate and publish data pipelines at scale
Pentaho
Pentaho offers comprehensive data integration and analytics platform.
MixDEM
MixDEM a web based ETL tools meant for Web integration, Data transformation and Mashup edition. It include MixDEM ETL Engine created using ZEND Framework, and MixDEM GUI Editor an AJAX IDE that enable developers to quickly and easily create applications.
rudderstack
Privacy and Security focused Segment-alternative, in Golang
GETL
ETL engine based on Groovy

Datapipe Files

Real-time, incremental ETL library for ML with record-level depend

Get an email when there's a new version of Datapipe