cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Learning Festival (Virtual): 15 January - 31 January 2025

Join us for the return of the Databricks Learning Festival (Virtual)! Mark your calendars from 15 January - 31 January 2025! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have el...

  • 67327 Views
  • 156 replies
  • 49 kudos
11-26-2024
Share Your Feedback in Our Community Survey

Your opinion matters! Take a few minutes to complete our Customer Experience Survey to help us improve the Databricks Community. Your input is crucial in shaping the future of our community and ensuring it meets your needs. Take the Survey Now Why p...

  • 472 Views
  • 0 replies
  • 0 kudos
Monday
Submit your feedback and win a $50 gift card!

Be among the first 20 people to share your experience with Databricks on G2 and receive a $50 gift card as our way of saying thanks. Why participate? Your feedback is invaluableIt helps us innovate and improve and takes less than 10 minutes to comple...

  • 366 Views
  • 0 replies
  • 3 kudos
Tuesday
Databricks Named a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems

We’re thrilled to share that Databricks has once again been recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems. This acknowledgement underscores our commitment to innovation and our leadership in the dat...

  • 1251 Views
  • 0 replies
  • 3 kudos
2 weeks ago
Milestone: DatabricksTV Reaches 100 Videos!

We are thrilled to announce that DatabricksTV, our growing video hub, has hit a major milestone: 100 videos and counting! What is DatabricksTV?DatabricksTV is a community-driven video hub designed to help data practitioners maximize the Databricks e...

  • 1559 Views
  • 1 replies
  • 4 kudos
12-11-2024
Announcing the new Meta Llama 3.3 model on Databricks

The Meta Llama 3.3 multilingual large language model (LLM) is a pretrained and instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimized for multilingual dialogue use cases and outperfo...

  • 1827 Views
  • 0 replies
  • 3 kudos
12-11-2024

Community Activity

David_Billa
by > New Contributor III
  • 33 Views
  • 2 replies
  • 1 kudos

Resolved! Explode function to flatten the JSON

I've the DDL as below. Create or replace table test ( prices ARRAY<STRUCT<Ord:STRING:,Vndr:STRING,Prc:STRING>> ) using delta location "path" Now I want to flatten the JSON and I've tried as below but it's throwing an error, "[UNRESOLVED.COLUMN.WITH_...

  • 33 Views
  • 2 replies
  • 1 kudos
Latest Reply
hari-prasad
Valued Contributor
  • 1 kudos

Hi @David_Billa ,You can use following from_json function in spark which can convert struct into individual column. Refer this link https://spark.apache.org/docs/3.4.0/api/python/reference/pyspark.sql/api/pyspark.sql.functions.from_json.html.Also, yo...

  • 1 kudos
1 More Replies
MuskanG
by > New Contributor
  • 3 Views
  • 0 replies
  • 0 kudos

Exam got suspended

Hi team,I attempted my Data Analyst Associate exam today, which was scheduled at 10:30 hrs. But my exam got suspended within the first 30 mins of the exam due to suspected eye movement error again and again.  I need help for this issue as I have take...

  • 3 Views
  • 0 replies
  • 0 kudos
bryan1
by > Visitor
  • 7 Views
  • 1 replies
  • 0 kudos

Customizing the UI of Streamlit App Template

Hi There,I used the Streamlit Chatbot template for Databricks Apps. I'm looking to customize the UI and wanted to make changes to the app.py file to do so. Even before I made any changes, I ran the file and I'm getting an error with databricks SDK an...

bryan1_0-1736647332600.png
  • 7 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @bryan1, Could you please advise what SDK version are you using?

  • 0 kudos
aonurdemir
by > New Contributor
  • 9 Views
  • 1 replies
  • 0 kudos

Is there a cluster option for dashboards?

Hi everyone,I do not want to use 4 DBU/h XS warehouse since I have very tiny data on the new startup. I want to create a minimal cluster and run it as the underlying SQL engine for my dashboard.Thanks.

  • 9 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Unfortunately no, as dashboards are part of the SQL service on the platform they are designed to work with SQL warehouses only, you can create Notebook dashboards that will be able to work with regular clusters but functionalities will be limited in ...

  • 0 kudos
atrix_computing
by > New Contributor II
  • 274 Views
  • 3 replies
  • 1 kudos

Customer Academy Labs for Self Paced

I was told by someone you could do labs in the self paced part of the databricks academy.  Right now I am on Demo: Creating and Working with a Delta Table in the Get Started with Databricks for Data Engineering course and I see no link to a lab anywh...

  • 274 Views
  • 3 replies
  • 1 kudos
Latest Reply
marcin2x4
Visitor
  • 1 kudos

Having same issue...

  • 1 kudos
2 More Replies
jeremy98
by > Contributor
  • 83 Views
  • 5 replies
  • 1 kudos

Migrating logic from Airflow DAGs to Databricks Workflow

Hello community,I'm planning to migrate some logics of Airflow DAGs on Databricks Workflow. But, I was facing out to some doubts that I have in order to migrate (to find the respective) the logic of my actual code from DAGs to Workflow.There are two ...

  • 83 Views
  • 5 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Yes, it is a good practice to create a separate task for the process_email functionality. This will help in modularizing your code and make it easier to manage and debug. Each task should ideally perform a single responsibility, and separating proces...

  • 1 kudos
4 More Replies
ismaelhenzel
by > Contributor
  • 351 Views
  • 2 replies
  • 8 kudos

DELTA LIVE TABLES - MATERIALIZED VIEW DOES NOT INCREMENT NOTHING !

I'm very disappointed with this framework. The documentation is inadequate, and it has many limitations. I want to run materialized views with incremental updates, but DLT insists on performing a full recompute. Why is it doing this? Here is the log ...

  • 351 Views
  • 2 replies
  • 8 kudos
Latest Reply
XP
Databricks Employee
  • 8 kudos

I'll try to clear up some misunderstanding surrounding the incremental load feature of materialized views:There isn't a feature to force materialized views to update incrementally. Instead, there is an optimizer called Enzyme that can selectively inc...

  • 8 kudos
1 More Replies
impresent
by > New Contributor
  • 26 Views
  • 1 replies
  • 0 kudos

Managed table storage not accessible in cloud storage

Hi All,I have create a new catalog in the unity catalog which has a cloud location for managed tables. but when accessing the location in Azure portal it denies me to see the files.I want to see all my data files ( parquet / JSON) in the storage acco...

  • 26 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Do you have full permissions on this storage location? Also please remember the best practices for Managed tables: You should not use tools outside of Databricks to manipulate files in managed tables directly. You should only interact with data files...

  • 0 kudos
SamGreene
by > Contributor II
  • 1372 Views
  • 2 replies
  • 0 kudos

String to date conversion errors

Hi,I am getting data from CDC on SQL Server using Informatica which is writing parquet files to ADLS.  I read the parquet files using DLT and end up with the date data as a string such as this'20240603164746563' I couldn't get this to convert using m...

  • 1372 Views
  • 2 replies
  • 0 kudos
Latest Reply
IdleGuys
New Contributor II
  • 0 kudos

@wt-asw I ran into the same problem, I have provided the table properties at the table declaration as below. Hope this helps!  CREATE OR REFRESH LIVE TABLE silver_table_name_mv TBLPROPERTIES ('delta.feature.timestampNtz' = 'supported') AS SELECT ....

  • 0 kudos
1 More Replies
Kayla
by > Valued Contributor
  • 29 Views
  • 1 replies
  • 0 kudos

GCP Serverless SQL Warehouse Tag Propagation?

I have a serverless SQL Warehouse with a tag on it that is not making it to GCP.We have various job and AP clusters with tags that I can see in GCP- trying to have everything tagged for the purpose of monitoring billing/usage centrally.Do serverless ...

  • 29 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

For non Serverless Warehouses are you able to see the tags making it to GCP, I am wondering if the issue might be that as the Serverless warehouse is managed in the Databricks cloud instead of your own cloud this could be the reason why this tags are...

  • 0 kudos
UM1
by > New Contributor
  • 537 Views
  • 4 replies
  • 1 kudos

drop or alter primary key constraint for a streaming table in delta live tables

I have a dlt streaming table with a primary key constraint defined through the schema definitions. I have redeployed the same dlt pipeline with the same target. When attempting to run the pipeline, I get the error,  ErrorClass=RESOURCE_ALREADY_EXISTS...

  • 537 Views
  • 4 replies
  • 1 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 1 kudos

The `RESOURCE_ALREADY_EXISTS` error you're encountering suggests that the primary key constraint `pk_constraint_name` already exists in the target delta table. This constraint may have been created during a previous deployment of the DLT pipeline or ...

  • 1 kudos
3 More Replies
manideep04d
by > New Contributor
  • 282 Views
  • 2 replies
  • 0 kudos

Creating Linked Service in ADF to link Databricks Community Edition

Using Current Community Edition I was Unable to create a linked service unable to generate/copy personal Access Token .Tried Installing ODBC but unable to set it properly. Any suggestions / Help would be highly Appreciated

  • 282 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

In community Edition workspaces the PAT token is not allowed, instead of this you might need to generate an Oauth token to be able to authenticate: https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.htmlYou can refer to the ODBC guide in page 19...

  • 0 kudos
1 More Replies
magrat08
by > New Contributor
  • 1824 Views
  • 3 replies
  • 0 kudos

Upload a file

Hi - I'm trying to upload a file, so that I can use the same in my notebook to try ML experiments with Databricks. From my workspace, I created a folder. But the option 'Create -> File' does not do anything. So not able to add any file. From a notebo...

  • 1824 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Can you try the approach mentioned in https://ganeshchandrasekaran.com/databricks-how-to-load-data-from-google-drive-github-c98d6b34d1b5 

  • 0 kudos
2 More Replies
charles898cabal
by > Visitor
  • 29 Views
  • 1 replies
  • 0 kudos

please enter your credentials...

Hi everyone,I'm trying to connect my local Jupyter Notebook environment to Databricks Community Edition, and I’ve followed the setup guide on the Databricks website. I’m attempting to use the mlflow library, but I’m facing an issue where the authenti...

  • 29 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

What is the process you are following to create the PAT token? In community Edition the PAT token is not allowed, instead you need to use Oauth token https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.html 

  • 0 kudos
Shreyas_24
by > Visitor
  • 20 Views
  • 1 replies
  • 0 kudos

Databricks rescheduling

Hi ,Can anyone from databricks team help in rescheduling the exam. I thought the exam was at  3 PM instead of 3 AM.Is there a possibility this can be reschedule to today anytime or tomorrow?   

  • 20 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

@Cert-TeamOPS can you assist on this request?

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Featured Event

Join Us for an Exclusive Databricks Community Event in San Francisco!

Thursday, January 23, 2025

View Event
Top Kudoed Authors
Read Databricks Data Intelligence Platform reviews on G2

Latest from our Blog

Deep Dive - Streaming Deduplication

In this article we will cover in depth about streaming deduplication using watermarking with dropDuplicates and dropDuplicatesWithinWatermark, how they are different. This blog expects you to have a g...

153Views 1kudos

Data Engineering SQL Holiday Specials

December is the most celebrated time of year in the Data Engineering calendar as we embrace the important holiday: change freeze season.  As we come back to the office to start our new projects, I wan...

1696Views 3kudos