Apache Spark 2.3 now on Databricks Runtime 4.0 Beta, Twitter donates Heron to Apache Software Foundation, new Blockchain-based platform to build AI apps, and more in today’s top stories around machine learning, deep learning, and data science news.
1. Apache Spark 2.3 Now Available on Databricks Runtime 4.0 Beta
Databricks announced the availability of Apache Spark 2.3.0 on Databricks as part of its Databricks Runtime 4.0 beta.
The Spark 2.3:
- Marks a major milestone for Structured Streaming by introducing low-latency continuous processing and stream-to-stream joins.
- Boosts PySpark by improving performance with pandas UDFs
- Runs on Kubernetes clusters by providing native support for Apache Spark applications.
The release extends new functionality to SparkR, Python, MLlib, and GraphX. It also focuses on usability, stability, and refinement, resolving over 1400 tickets.
For additional features and other information, read the Spark 2.3 release notes.
2. New Blockchain-Based Platform to Collectively Build AI Apps
Dbrain, a new project built on the Ethereum Blockchain leverages smart contracts to develop a simple tool that allows everyone to label and validate data in exchange for cryptocurrency. Dbrain introduces a platform that targets businesses and data scientists that need the data to develop AI solutions.
By building smart contracts on Ethereum, Dbrain plans to use its internal protocols to solve fundamental AI-based development, execution, and adoption challenges which include:
- Dataset quality
- Trust and security
- Infrastructure costs
It aims to have a complete AI production line integrated with its platform. Thus it combines labeling functionality and ensures that payment validation is transparent. It also aims to provide customized AI solutions within a single product.
To know more about the challenges in detail, visit Dbrain’s blogpost.
3. Now on GitHub: The Autonomous Driving Cookbook from Microsoft as Jupyter Notebooks
The Autonomous Driving Cookbook from Microsoft is now available on GitHub.
The cookbook is an open source collection of scenarios, tutorials, and demos to help you quickly onboard various aspects of the autonomous driving pipeline.
It is an ongoing project developed and maintained by the Deep Learning and Robotics chapter of Microsoft Garage, the team that helped develop the recent expansion of AirSim to include cars for autonomous driving research.
Tutorials in the cookbook are presented as Jupyter notebooks, making it very easy for you to download the instructions and get started without a lot of setup time. To help this further, wherever needed, tutorials come with their own datasets, helper scripts and binaries.
Read more of this in detail at Microsoft + Open Source Blogpost here.
4. Tenet Partners Launches Data Analytics Platform
Tenet Partners announced CoreBrand® Data Science, a new business unit leveraging the power of predictive analytics and data science to transform how corporations and capital markets can generate value from their brands. Tenet Partners help the C-suite to drive positive business outcomes by using a combination of research that underpins the CoreBrand® Index and advanced analytics.
Read the official press release for detailed information about this launch.
5. Twitter donates Heron to Apache Software Foundation
Twitter announced that it is donating Heron to the Apache Incubator where the community will continue to grow and thrive under the guidance of the Apache Software Foundation.
Heron is a real-time analytics platform developed by Twitter to reliably process billions of events generated at Twitter every day.
Open-sourced in 2011, it is the next generation distributed streaming engine that was built to be backwards compatible with Apache Storm. It was built to improve Twitter’s developer and operational experiences with Storm and introduced a wide array of architectural improvements and native support for Apache Aurora.
Heron has become Twitter’s primary streaming system, reliabily powering all of Twitter’s real-time analytics and running hundreds of development and production topologies deployed on thousands of nodes.
For more, read Twitter’s official announcement.