As Lead Architect within Engineering at Optima Partners, I recently attended the Databricks Data+AI World Tour in London. The event provided valuable insights into artificial intelligence, particularly generative AI and the future of AI, and its impact across various industries. I’d like to share my key takeaways and observations from this enlightening event.
The Rise of Generative AI
The most striking revelation from the conference is the undeniable growth of generative AI. This technology is not just a passing trend; it’s becoming increasingly prevalent across diverse sectors, revolutionising traditionally complex and resource-intensive processes.
Take Rolls-Royce, for instance. They’re now using generative AI in their design and development processes for both automotive and aerospace engines. This technology allows engineers to produce design images without relying on costly simulation software, potentially streamlining their development cycle and reducing costs.
In a completely different industry, Sega is harnessing generative AI for game design. By utilising a vast library of over 100,000 historical Sega assets, designers can now create new assets based on text descriptions. This application of AI not only speeds up the creative process but also ensures a consistent style that aligns with Sega’s rich gaming heritage.
Pushing the Boundaries of AI and ML
Databricks itself has made significant strides in expanding their machine learning and generative AI offerings. Through strategic acquisitions, they’ve strengthened their capabilities in these areas. A key focus has been on unifying data formats into a single layer, eliminating the need for trade-offs when choosing between different formats.
This approach aligns with their concept of “democratised AI”, which they’re integrating into every part of the data lifecycle. One of their exciting new features is the AI/BI Genie. This tool enables businesses to generate dashboards from their Databricks-hosted data, complete with business context and definitions. It’s a prime example of how AI can make data more accessible and actionable for organisations.
The Open-Source AI Revolution
It’s worth noting that open-source generative AI models are rapidly improving, now rivalling some of the best paid models. Databricks is fully embracing this trend, supporting open-source models from vendors like Meta and optimising their use within the Databricks platform. This development could significantly lower the barrier to entry for companies looking to leverage advanced AI capabilities.
Tackling Data Governance and Discovery
A significant portion of the conference focused on data governance architecture, search, and discoverability. As AI becomes more prevalent, companies need robust systems to track how their data assets are being used and understand the context surrounding them. This includes identifying personal information and data product ownership.
There’s a growing emphasis on discovering information about data, rather than just searching for the data itself. Databricks is working to simplify these complex challenges for users, leveraging AI to address issues around searchability and discoverability while maintaining proper data governance. This approach could potentially solve some of the hardest problems in data management, making it easier for organisations to maintain compliance while still deriving value from their data.
The Future of AI and Data
The future of AI and data management present exciting opportunities for innovation and efficiency across various industries. From streamlining design processes to revolutionising game development, from democratising data access to solving complex governance issues, the potential applications are vast and varied.
If you’d like to discuss how these developments might impact your organisation or explore ways to leverage these technologies, please don’t hesitate to get in touch through my personal LinkedIn here, or via the contact form.
