Databricks provides robust access control mechanisms that can be applied at both column-level and row-level to secure sensitive data. Through dynamic views, you can apply Access Control Lists (ACLs) to restrict visibility based on user privileges. This ensures that users with sufficient access can view all the data, while restricted users are presented with redacted or filtered data, depending on the predefined rules set at the view level.
In scenarios where sensitive information such as email addresses, first names, and last names exists, managing access to these fields is critical for data privacy. Databricks enables this through dynamic views, helping organizations enforce data governance policies and comply with regulatory requirements such as GDPR.
What is Access Control?
Access Control refers to the mechanisms that restrict access to data and resources based on predefined policies. Access control is essential for maintaining data security and privacy in the context of databases and data platforms like Databricks.
Access control can be applied at various levels, including:
User-Level Access Control: Grants permissions to individual users based on their roles and responsibilities within the organization. This often involves authenticating users and authorizing them to access specific resources.
Group-Level Access Control: Simplifies management by allowing administrators to assign permissions to groups of users rather than individually. This approach makes it easier to manage large teams and enforce consistent security policies.
Column-Level Access Control: Provides a mechanism to restrict access to specific columns in a dataset. Sensitive information such as personal identifiers (e.g., email addresses, names) can be hidden from unauthorized users while still allowing them to access other relevant data.
Row-Level Access Control: Allows the application of filters on the data, restricting access to certain rows based on user roles or attributes. For example, only allowing users to view records that pertain to their geographic region.
What are Dynamic Views?
Dynamic Views in Databricks refer to a type of database view that can change based on the current user’s access level and privileges. Unlike static views, which present a fixed set of data, dynamic views use SQL logic to filter or modify the output depending on the user’s group membership or roles.
Key features of dynamic views include:
Real-Time Data Security: By incorporating user roles and conditions into the view definition, dynamic views ensure that data is served in real-time based on the user’s access rights. This means sensitive information can be redacted or filtered automatically without needing to create separate views for each user group.
Simplified Data Management: Dynamic views streamline the management of data access by allowing a single definition to serve multiple user roles. This reduces the complexity of creating and maintaining multiple views for different user groups.
Flexible Data Presentation: Dynamic views can present different data sets to different users based on their roles, enhancing user experience while ensuring data privacy and compliance with governance policies.
Column-Level ACLs:
For column-level security, Databricks allows you to redact sensitive columns for unauthorized users. For example, the following dynamic view will hide customer names and email addresses for users who are not part of the “demo_admin” group:
- Create a table with sensitive information.
2. View the customers_orders data
3. Column level ACLs:
In this example:
- Non-members will see the value “REDACTED” in place of these sensitive fields.
- Members of the demo_admin group can see the first_name, last_name, and email in plain text.
This setup ensures that only users with sufficient privileges can view personally identifiable information (PII), while all other users are provided with redacted data to maintain privacy.
Row-Level ACLs:
In addition to column-level security, Databricks supports row-level access control. By using dynamic views and applying filtering conditions in the WHERE clause, access to specific rows of data can be restricted based on user roles.
For instance, the following view restricts data visibility to only show USA-based orders after a specific date for unauthorized users, while admin users get full access to the entire dataset
In this example:
- Non-members only see rows where the country is “USA” and the date is after “2022-01-01”. This kind of filtering is particularly useful in scenarios where data access must be limited based on geographic or temporal restrictions.
- Members of the “demo_admin” group see all records, as the CASE statement returns TRUE for them.
How to Implement Access Control in Databricks:
Column-Level Control: Use the CASE statements inside SELECT queries to conditionally display redacted or real data based on group membership.
Row-Level Control: Use the WHERE clause in views to apply row-level filters dynamically. This can be based on conditions such as the user’s group, country, date, or other parameters.
Admin Console: Manage users and groups through the Databricks Admin Console, where you can assign users to groups like demo_admin. You can configure and manage group memberships to control data access.
Conclusion:
Databricks’ dynamic views provide a flexible and powerful mechanism for implementing fine-grained data security at both the column and row levels. By using dynamic ACLs, organizations can ensure that PII is protected and only exposed to authorized users. This approach also helps meet compliance requirements, such as GDPR, by ensuring sensitive information is not inadvertently shared.
Through tools like the is_member( ) function and dynamic views, Databricks allows for scalable data privacy controls, enabling teams to safely share data while maintaining strict security controls. Leveraging these controls not only enhances data governance but also builds trust with stakeholders by ensuring that sensitive customer information is handled securely.
For more information on PII data and its security, please refer to this blog: Ensuring Secure Storage of PII Data
For More Details, Diggibyte Technologies Pvt Ltd has all the experts you need. Contact us Today to embed intelligence into your organization.
Author: Marripudi Haritha Sumanjali